Imaging signal extraction apparatus and methods of using same

ABSTRACT

An imaging signal extraction apparatus comprising: an interface; a processing device, the processing device operatively coupled to the interface; and a computer readable medium comprising instructions that, when executed by the processing device, perform operations comprising: a) generating a two-dimensional image from imaging information obtained from the interface, thereby estimating ballistic component of the imaging information; b) generating a three-dimensional image by remapping the two-dimensional image; c) identifying a candidate object in the three-dimensional image; d) obtaining an estimated spatial forward model of the candidate object by mapping the three-dimensional image of the candidate object with a point-spread-function associated with the imaging apparatus; e) obtaining background-corrected data by using the estimated spatial forward model of the candidate object and estimated temporal components; and f) iteratively updating the estimated spatial forward model and estimated temporal components until convergence is reached for the candidate object, thereby extracting the signal information.

This application is a U.S. National Phase application of InternationalApplication No. PCT/US18/33417, filed May 18, 2018, which claims thebenefit of U.S. Provisional Application No. 62/640,377, filed Mar. 8,2018, and U.S. Provisional Application No. 62/508,604, filed May 19,2017; the disclosures of which are incorporated herein by reference intheir entireties.

The invention was made with government support under contract no.D16PC00002 Intelligence Advanced Research Projects Activity (IARPA)awarded by the Department of Interior/Interior Business Center(DoI/IBC). The invention was also made with government support undergrant no. DBI-1707408 awarded by the National Science Foundation. Thegovernment has certain rights in the invention.

BACKGROUND OF THE INVENTION Field

The disclosed embodiments relate to extracting signals from time seriesrecordings, including, for example, imaging recordings, e.g., imagingrecordings in a scattering medium.

Related Art

Understanding multi-scale integration of sensory inputs and theemergence of complex behavior from global dynamics of large neuronalpopulations is a fundamental problem in current neuroscience. Onlyrecently, the combination of genetically encoded Calcium (Ca²⁺)indicators (GECIs)₁ and new optical imaging techniques has enabledrecording of neuronal population activity from entire nervous systems ofsmall model organisms, such as C. elegans _(2,3) and zebrafishlarvae_(4,5), at high speed and single-cell resolution. However,single-cell resolution functional imaging of large volumes at high speedand great depth in scattering tissue, such as the mammalian neocortex,has proven challenging.

A major limitation is the fundamental trade-off between serial andparallel acquisition schemes. Serial acquisition approaches, such asstandard two-photon scanning microscopy (2PM)₆, in which spatialresolution is determined by the 3D locations of the excitation, providerobustness to scattering and signal crosstalk in the emission path, asthe emitted fluorescence is integrated on a point detector. Thiscapability has made 2PM the standard method for deep tissue imaging₇.However, this has been achieved at the expense of temporal resolutionsince the excitation spot needs to be scanned in 3D. More recently, anumber of approaches have been developed to alleviate this restriction₈at the cost of increased complexity, e.g., by scanning faster usingacousto-optic deflectors₉, remote focusing using mechanical actuators₁₀or acousto-optical lenses₁₁, temporal or spatial multiplexing₁₂₋₁₄, byselectively addressing known source positions by random accessscanning₁₅₋₁₇, or by sculpting the microscope's point spread function(PSF) in combination with a more efficient excitation scheme₁₈.

In contrast, parallel acquisition schemes, such as wide-fieldepi-fluorescence microscopy, light-sheet microscopy_(19,20,5), includingmulti-view light-sheet techniques₂₁ and swept confocally aligned planarexcitation₂₂, wide-field temporal focusing₂, and holographicapproaches₂₃₋₂₅ can improve temporal resolution. Typically, in thesemethods, multiple regions or the entire sample are excitedsimultaneously and the fluorescence light is detected using 2D sensorarrays. Typically, however, light scattering mixes fluorescence signalsoriginating from distinct neurons and degrades information about theirlocations. Thus, parallel acquisition schemes have been mostly limitedto highly transparent specimens or to the most superficial regions ofscattering tissues, such as the mammalian cortex.

SUMMARY OF THE INVENTION

The embodiments disclosed herein include an imaging signal extraction(e.g., demixer), apparatus which includes an imaging apparatusinterface, a processing device, and a computer-readable medium. Theimaging apparatus can be any apparatus that maps a three-dimensionalsample volume location onto a two-dimensional sensor location in aspecific manner. An example of such a device is a light-fieldmicroscope. The processing device is operatively coupled to the imagingapparatus interface. The computer readable medium includes instructionsthat, when executed by the processing device, perform operationsincluding (a) generating a two-dimensional image (e.g., two-dimensionalstandard deviation image), from imaging information obtained from theimaging apparatus interface, thereby estimating ballistic component ofthe imaging information, (b) generating a three-dimensional image (i.e.,3D volume) by remapping (e.g., deconvolving) the two-dimensional image,(c) identifying a candidate object in the three-dimensional image, (d)obtaining an estimated spatial forward model of the candidate object bymapping (e.g., convolving) the three-dimensional image of the candidateobject with a point-spread-function associated with the imagingapparatus, (e) obtaining background-corrected data by using theestimated spatial forward model of the candidate object and estimatedtemporal component, and (f) iteratively updating the estimated spatialforward model and estimated temporal components until convergence isreached for the candidate object, thereby demixing the signalinformation.

In one embodiment, before operation (a), background information obtainedby the imaging apparatus may be subtracted, using the imaging apparatusinterface. The background information may be background fluorescenceobtained from a light-field microscope, and the subtraction of thebackground information may include applying rank-1-matrix factorization.Operation (a) may include determining the standard deviation of a timeseries of camera frames, and operation (b) may include using apoint-spread-function associated with the imaging apparatus. The pointspread function can be numerically simulated or experimentally obtained,and can be a ballistic or non-ballistic spread-function. Beforeoperation (b), the two-dimensional standard deviation image may bethresholded to exclude residual background activity, and operation (b)further may include reducing reconstruction artefacts by incorporatingtotal-variation and sparsity constraints into the remapping (e.g.,deconvolution).

Reducing reconstruction artefacts may include applying the equationx_(n+1)=x_(n)(P^(T)y/P^(T) P y+λ1_(dim(x))), wherein x represents avolume estimate, 1_(dim(x)) represents a vector of ones with samedimension as x, P represents the point-spread-function, λ representsweight of a sparsity-encouraging term, and y represents the backgroundsubtracted raw data. Operation (c) may include using spatialsegmentation to suppress spatial frequencies incompatible with objectshapes. The spatial segmentation may include applying a bandpass filterto the three dimensional image, thresholding to exclude backgroundartefacts, and applying a local maximum search algorithm. Operation (d)of mapping (e.g., convolving) the three-dimensional image of thecandidate object with the point-spread-function associated with theimaging apparatus may include producing a sparse non-negative p×n matrixS_(i), wherein n is the number of object candidates, p is the number ofpixels and i is the iteration number, wherein S₀ is the initial spatialforward model of the candidate object. Operation (e) may includegenerating a p×t matrix Y using the matrix product of S₀ and T₀, whereinT_(i) is a non-negative n×t matrix of temporal components, wherein t isthe number of time steps in the recording. T_(i) may be obtained byiteratively applying an adapted Richardson-Lucy-type solver with asparsity constraint. Iteratively updating the estimated spatial forwardmodel and estimated temporal components may include (i) obtaining anupdated estimated S_(i), while keeping estimated T_(i) constant, (ii)obtaining an updated estimated T_(i), while keeping estimated S_(i)constant, and (iii) iteratively repeating operations (i) and (ii) untilconvergence is reached for the object candidate. The candidate objectmay be a neuron.

In addition to enabling efficient signal extraction in a scatteringmedium (e.g., scattering tissue) and providing increased temporal andspatial fidelity in semi-transparent specimens, a key advance of thedisclosed embodiments is a dramatic reduction in computational costcompared to previous image reconstructions (e.g., image reconstructionsfor LFM) and post-processing by three orders of magnitude. This enablesa range of qualitatively new applications, including real-timewhole-brain recording, closed loop interrogation of neuronal populationactivity in combination with optogenetics and behavior, and theapplication of advanced machine learning techniques to analysis of data.

In another embodiment, the imaging signal extraction apparatus includesan imaging apparatus interface, a processing device operatively coupledto the imaging apparatus interface, and a computer readable mediumcomprising instructions that, when executed by the processing deviceperform operations. The operations include generating a two-dimensionalimage from imaging information obtained from the imaging apparatusinterface, thereby estimating ballistic component of the imaginginformation, generating a three-dimensional image by remapping thetwo-dimensional image, identifying a candidate object in thethree-dimensional image. obtaining an estimated spatial forward model ofthe candidate object by mapping the three-dimensional image of thecandidate object with a point-spread-function associated with theimaging apparatus, obtaining background-corrected data by using theestimated spatial forward model of the candidate object and estimatedtemporal components, and iteratively updating the estimated spatialforward model and estimated temporal components until convergence isreached for the candidate object, thereby extracting the signalinformation. The imaging apparatus interface includes hardware developedusing a Miniscope platform, an implanted endoscopic GRIN relay, asensor, and a microlens array. The microlens array is aligned andmounted in close proximity to the sensor such that a back focal planeand a sensor plane coincide. The microlens array may be disposed in anoptical path of an image plane one focal length away from the sensor.The apparatus may also include a holding member configured to hold thesensor. The holding member may be elongated by 2.7 mm when compared withthe Miniscope design.

In one embodiment, the invention provides a method of extracting imagingsignals. The method comprises using an imaging apparatus interface thatis operatively coupled to a processing device. The processing deviceperforms the following operations: a) generating a two-dimensional imagefrom imaging information obtained from the imaging apparatus interface,thereby estimating ballistic component of the imaging information; b)generating a three-dimensional image by remapping the two-dimensionalimage; c) identifying a candidate object in the three-dimensional image;d) obtaining an estimated spatial forward model of the candidate objectby mapping the three-dimensional image of the candidate object with apoint-spread-function associated with the imaging apparatus; e)obtaining background-corrected data by using the estimated spatialforward model of the candidate object and estimated temporal components;and f) iteratively updating the estimated spatial forward model andestimated temporal components until convergence is reached for thecandidate object, thereby extracting the signal information.

Other embodiments will become apparent from the following detaileddescription considered in conjunction with the accompanying drawings. Itis to be understood, however, that the drawings are designed as anillustration only and not as a definition of the limits of any of theembodiments.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1—Seeded iterative demixing of light-field recordings in scatteringtissue. Illustration of key steps in the Seeded Iterative Demixing (SID)algorithm.

FIG. 2—Video-rate volumetric Ca²⁺ imaging in mouse hippocampus.Schematics of the hippocampal window preparation, indicating corpuscallosum (CC), and region of hippocampus proper Cornu Ammonis (CA1, CA3)and dentate gyrus (DG), the rectangle above CA1 indicates theapproximate imaging volume.

FIG. 3—Statistical analysis of SID neuron detection and signalextraction performance based on simultaneous 2PM-SID recordings; (a)neuron detection scores versus depth as achieved by SID (green traces),in comparison to scores achieved by the analysis package CaImAn appliedto the 2PM data (blue traces), both evaluated with respect to a groundtruth; (i) sensitivity score (ratio of number of detected to actualneurons); (ii) precision score (ratio of number of true positives to sumof true and false positives); (iii) F-Score (harmonic mean ofsensitivity and precision) n=4; (b) comparison of SID extracted signalsto ground truth; (i) correlation means versus depth and (ii) histogramof correlation coefficients of SID signals and their ground truthcounterparts, shown for one example; (iii) examples of two pairs of SID(green) and corresponding ground truth (red) signal pairs and theirrespective correlation coefficients; (iv) ratio of SID-signals withcorrelation to ground truth of less than <0.5 versus imaging depth.

FIG. 4 is a block diagram of at least a portion of an exemplary machinein the form of a computing system that performs methods according to oneor more embodiments disclosed herein.

FIG. 5 shows the Head-mounted miniature Light Field Microscope(MiniLFM). Explosion (left) and section drawing (right) of MiniLFM areshown. Some parts are rendered transparently for visual clarity.

FIG. 6 shows a rendering of a MiniLFM MLA-to-sensor alignment jig. Foraligning the MLA to the sensor chip, a pair of custom 4-finger holders(silver cylindrical slotted parts, center left) was designed that can betightened using hose clamps (not shown). One clamp holds the MLA (notvisible, occluded by clamp) and is mounted statically on a post/postholder (leftmost part). The other clamp holds the sensor (turquoiserectangle) and is itself held by a 6-axis kinematic mount (ThorlabsK6XS) for adjusting tip, tilt and rotation, and lateral position. Thekinematic mount is attached to a 3-axis linear stage assembly (ThorlabsPTA3A/M) for adjusting MLA-to-sensor distance as well as for convenientcoarse adjustment of lateral position.

FIG. 7 includes graphs showing a comparison of animal agility whenwearing no device, Miniscope, and MiniLFM. Quantification of animalagility is shown from recordings of behavior on a linear track, aftercompletion of training. Three mice; one trial under each condition peranimal and day, for three consecutive days, resulting in a total n=27trials. Trial duration: 10 minutes. Inter-trial break: 1 hour. Widehorizontal bars indicated mean, error bars are s.e.m. Data point colorindicates animal. (a) Average walking speed. ns, not significant byone-way ANOVA. (b) Distance travelled per trial. ns, not significant byone-way ANOVA. (c) Number of stops made during trial. ns, notsignificant; *, significant at p<0.05 by one-way ANOVA (p=0.011).

FIG. 8 is a sketch of an experimental setup used for simultaneous2PM+MiniLFM/SID recordings.

It is to be appreciated that elements in the figures are illustrated forsimplicity and clarity. Common but well-understood elements that areuseful or necessary in a commercially feasible embodiment are not shownin order to facilitate a less hindered view of the illustratedembodiments.

DETAILED DESCRIPTION OF THE INVENTION

The disclosed embodiments relate to extracting imaging signals from timeseries recordings. A time series is a series of data points indexed intime order. An example of extraction of imaging signals is demixing ofsignals from imaging recordings, in particular imaging recordings in ascattering medium.

The imaging signal extraction apparatus of the disclosed embodiments (1)exploits the high resolution spatial information contained in remnantballistic light, as well as extract directional information fromscattered light, (2) incorporates the particular imaging apparatus'point spread function (PSF) and the effects of scattering, and (3)extracts (e.g., demixes) signals from close lying sources within avolume (e.g., demixes the effects of scattering) by utilizing both thespatial and temporal information present in the imaging data withoutrequiring further assumptions on source positions or signalcharacteristics.

In one embodiment, an imaging signal extraction apparatus is provided.The apparatus includes an apparatus interface, a processing device,which is operatively coupled to the apparatus interface, and a computerreadable medium including instructions, that, when executed by theprocessing device, perform operations to extract (e.g., demix) signalinformation.

The imaging signal extraction apparatus can be any apparatus which mapsthree-dimensional (3D) images onto a two-dimensional (2D) sensor array,in particular, those that use parallel acquisition schemes. Examples ofsuch imaging apparatus include a light-field microscope (LFM),wide-field epi-fluorescence microscope, light-sheet microscope,including multi-view light-sheet techniques and swept confocally alignedplanar excitation, wide-field temporal focusing and holographicapproaches. Typically, in these methods, multiple regions or the entiresample are excited simultaneously and the fluorescence light is detectedusing 2D sensor arrays.

Imaging with Light-Field Microscope

In a preferred embodiment, the imaging apparatus is the LFM. The LFMachieves extremely high volume acquisition rates (limited only by GECIresponse dynamics and camera frame rate) at large fields-of-view byefficiently mapping 3D volumetric information onto a 2D sensor array,wherein a microlens array is placed in the image plane of a microscope,and a camera in the focal plane of the microlens array. This results ina spatially varying point-spread function (PSF), which encodes both thespatial and angular coordinates of incident light rays into 2D patternson the sensor. The full 3D information is captured by a single cameraexposure and retrieved offline by computational remapping (e.g.,deconvolution) of the raw images.

Among parallel acquisition techniques, Light Field Microscopy(LFM)_(4,26-29) is a particularly simple yet powerful approach to highspeed volumetric Ca²⁺ imaging in small semi-transparent model systems,such as C. elegans and zebrafish larvae.₄ LFM stands out from competingimaging methods by not requiring any time-consuming scanning of theexcitation beam to collect 3D information. Moreover, in contrast tomethods based on two-photon excitation, LFM does not require expensiveand complex ultrafast laser systems and is not prone to sample heatingand nonlinear photo-damage.

LFM achieves extremely high volume acquisition rates (limited only byGECI response dynamics and camera frame rate) at large fields-of-view byefficiently mapping 3D volumetric information onto a 2D sensor array,wherein a microlens array is placed in the image plane of a microscope,and a camera in the focal plane of the microlens array. This results ina spatially varying point-spread function (PSF), which encodes both thespatial and angular coordinates of incident light rays into 2D patternson the sensor. The full 3D information is captured by a single cameraexposure and retrieved offline by computational remapping (e.g.,deconvolution) of the raw images._(4,27)

The information that LFM collects is vectorial and redundant innature._(29,30) In LFM, both the positions and directions of incidentlight rays are recorded, and the ensemble of all rays emitted by a pointsource and transmitted by the optical system forms a highly specific PSFpattern on the sensor.

However, conventional frame-by-frame reconstruction of LFM images_(4,27)largely fails at harvesting the potential robustness inherent to LFMdata, in addition to being highly computationally resource intensive.

On average, after propagating for the characteristic distance of onescattering length (˜50-100 μm for visible light in the cortex₇), some34% of incident photons still travel in their original direction, whichare referred to as “ballistic photons”; whereas the remaining photos aredeflected by a random scattering angle. In brain tissue, the probabilitydistribution of scattering angles, a Henyey-Greenstein distribution withanisotropy parameter g≈0.97, is not uniform, but strongly peaked aroundthe forward direction. Thus, information on the original direction ofthe scattered photons is retained for several scattering lengths₇, butthis information is blurred and spread into a cone-shaped region aroundthe remaining ballistic photons.

In conventional wide field imaging, similar to the effect of a defocus,scattering causes image features to appear blurred and overlapping,rendering demixing a highly ill-posed mathematical problem. In contrast,in LFM, in the absence of scattering, a source located below or abovethe focal plane results in sharp and specific patterns on the sensorthat encode both positional and angular information about the incidentlight field. In a scattering medium, the scattered photons in LFM aredistributed over many sensor pixels around those illuminated by theballistic photons. Notably, any directional information retained in thescattered rays manifests itself as a direction-specific gradient in theintensity distribution of scattered light, wherein a ballistic peak andgradient are due to scattering, as indicated by arrow. In the absence ofscattering, deconvolution of the raw LFM images using a numericallysimulated, ballistic PSF_(4,27) allows nearby neurons to be resolved,and to faithfully recover their respective temporal signal. In thepresence of scattering, however, the same image reconstruction methodincreasingly fails to faithfully recover the signals of nearby neuronswith increasing depth due to crosstalk. In addition, scattered lightleads to the emergence of reconstruction artefacts, and erroneousassignment of brightness to a diffuse background component. Together,these effects render signal extraction in scattering tissue usingpreviously established deconvolution schemes_(4,27) a non-trivial task.

However, since some directional information is retained in the scatteredlight field and recorded by LFM, a more robust signal extraction fromraw LFM data is necessary. Methods based on spatial imagesegmentation_(31,32) cannot be expected to yield useful results in theabsence of clear contours. A more commonly used approach for extractingneuronal signals from (predominantly 2PM-based) Ca²⁺ activity movies isbased on Independent Component Analysis (ICA)₃₃. ICA can perform wellwhen neurons are fairly well-separated. However, when the recordedimages of a set of neurons overlap spatially or if their activities arestrongly correlated, ICA often fails to demix these sources correctly₃₄.Methods based on non-negative, sparse and otherwise constrainedspatio-temporal matrix factorization₃₄₋₃₇ surpass ICA in demixingcapability for closely packed neurons, especially when spatial andtemporal constraints are incorporated₃₄. On the practical level,however, these methods typically require appropriate initialization ofspatial components with high accuracy for a robust and quick convergenceof the algorithm. Furthermore, currently available implementations donot include information on the imaging system such as its PSF, let alonestochastic and unspecific processes such as scattering.

Neuronal Imaging with Head Mounted Apparatus

Capturing neuronal dynamics volumetrically at high speed and single cellresolution in freely behaving rodents has remained a major outstandingchallenge in neuroscience. The combination of Light field microscopy(LFM) and Seeded Iterative Demixing (SID) enables realization of ascalable high-speed volumetric calcium imaging method for applicationsin the strongly scattering mammalian cortex.

A miniaturized head-mounted light-field microscope (“MiniLFM”) wasdesigned and built, which in combination with the SID algorithm enablescalcium imaging within a volume of ˜600×600×350 μm at 16 Hz volume rate,thereby capturing the dynamics of ˜530 neurons per imaging session inthe hippocampus of freely moving mice. Performance of the MiniLFM andoptimized SID algorithm was proven by showing extraction and assignmentof neuronal activity traces as deep as 345 μm from the surface of theimplanted GRIN objective lens.

Another key feature is a unique rigid hardware design and head-mountingassembly that minimizes motion artifacts, while a dedicated processingpipeline detects any residual motions in the raw imaging data withoutthe need for additional motion sensors and corrects for these to ensurethat SID-processing remains unaffected. Moreover, the pipeline trains amodel for the underlying firing rate and calcium indicator responsedynamics and provides a robust estimate of the firing rate, even for themotion-affected frames.

To understand the highly integrated cognitive processes in mammals, aswell as the neuronal basis of complex and ethologically relevantbehavior, fast, depth-penetrating volumetric imaging techniques are usedthat are compatible with free behavior and social interaction. Beforethe current subject matter, all existing volumetric Ca²⁺ imagingtechniques capable of extracting information from the mammalian or avianbrain required head fixation. A number of portable, head-mountedminiature microscopes have been developed that enable recording fromfreely moving animals_(20A-24A), however, none of these is capable ofvolumetric imaging. Initial designs of head-mounted fluorescence imagingdevices_(20A,25A,26A) used optical fibers for light delivery from lasersources to implement confocal or two-photon excitation, while forfluorescence detection, readout via individual optical fibers_(27A) aswell as fiber bundles_(21A) has been explored. Deep brain structures areaccessible in a widefield configuration when implanted endoscopicelements such as gradient index (GRIN) rod lenses_(27A) are used. Morerecently, single-photon, wide-field miniature microscopes(“Miniscopes”)_(22A-24A,28A) have been built that have enabled long-termrecording of hippocampal place cells_(28A), and studying the encoding oflocomotion-relevant information in the dorsal striatum_(24A) as well asthe role of shared neural ensembles in the association of distinctcontextual memories_(23A). These studies highlight the importance ofneuronal recording during unrestrained behavior to uncover the neuronalbasis of ethologically relevant and complex behaviors.

One embodiment of the disclosed subject matter overcomes theaforementioned limitations by combining head-mounted miniaturemicroscope (“Miniscope”) technology_(23A) with Light FieldMicroscopy-based (LFM)_(3A,29A) detection and a computational strategybased on a constrained matrix factorization approach (Seeded IterativeDemixing, “SID”)_(4A) that offers increased robustness to lightscattering. LFM allows capturing volumetric information in a singleexposure of a 2D image sensor, while SID extends the reach of LFM intothe scattering mammalian brain_(4A). The disclosed subject matterprovides a miniaturized head-mounted SID microscope using LFM hardware(“MiniLFM”), which allows Ca²⁺-imaging within a volume of ˜700×600×360μm at 16 Hz volume rate, thereby capturing the dynamics of ˜810 neuronsper imaging session at near-single-cell resolution in the hippocampus offreely moving mice. The SID algorithm_(4A) allows the extraction andassignment of neuronal activity traces as deep as 360 μm from thesurface of implanted GRIN objective lenses.

The hardware design of the MiniLFM differs from typical LFM designs intwo important aspects: First, the MiniLFM design (FIG. 5) leverages theopen-source Miniscope platform_(23A), which is optimized for minimalweight, simplicity of operation, and compatibility with implantedendoscopic GRIN relays to reach deep brain structures. Second, thetypical configuration of relaying the focal plane of the microlens array(MLA) onto the camera sensor plane has been replaced with an approach inwhich the microlens array is aligned and mounted in close proximity tothe sensor, such that the MLA back focal plane and the sensor planecoincide (FIG. 5). A major advantage of this approach is that byincorporating only one additional optical element, the microlens array,the overall weight of the MiniLFM is kept minimal.

The alignment strategy allows for accurate, quantitative optimization ofMLA orientation and position relative to the image sensor prior tofixation. Exact alignment is critical, since good overlap between thenumerically simulated point-spread function (PSF) of the system and thephysical PSF is required for recovering the volumetric data from the 2Draw image by deconvolution_(3A,30A).

The microscope achieves a lateral resolution of 80 line pairs permillimeter, which corresponds to a spot size of ˜6 μm, and ˜30 μm axialresolution. However, in the presence of scattering, the opticalresolution is not generally what quantifies the limits fordiscriminating neurons. The actual spatial discriminability is furtherdetermined by factors, such as the amount of spatial overlap of theneurons' scattered spatial footprints on the sensor, in combination withthe similarity of their activity in time. The minimum distance betweentheir centroids, at which two neurons can be robustly demixed, is calledherein “the discrimination threshold.” In one embodiment, this thresholdwas found to be ˜15 μm.

The head-mounted module is portable by an adult mouse, allowing it tomove freely in an arena. Video shows adult mouse behaving and movingspontaneously for 50 s in arena. MiniLFM is screw-clamped into abaseplate that had been glued to the skull, and centered on an implantedGRIN objective lens. The data cable is suspended from an arm above thecenter of the arena. The potential effect of device weight on animalagility was characterized by recording and quantifying the animal'sbehavior on a linear track for three conditions: wearing a standardMiniscope, a MiniLFM, or no device. While, as expected, a slight trendin reduced agility from animals without a device to animals wearing theMiniscope, and from animals wearing a Miniscope to animals wearing aMiniLFM could be observed, no significant difference in distancetravelled, number of stops, or the average speed, between MiniLFM andthe Miniscope was found.

Next, the performance of the MiniLFM was verified by recordingspontaneous volumetric activity of hippocampal CA1 neurons in freelymoving mice. While the raw MiniLFM frames appear highly blurred on thecamera and do not allow the identification of individual neurons,applying the SID algorithm allows for clear extraction of neuronalpositions and corresponding activity time series in the CA1 pyramidaland Stratum radiatum layers down to a depth of 360 μm. Moreover, theability of the method to perform volumetric recording reveals the shapeof the pyramidal layer more clearly through the 3D rendering of therecoding volume. Neurons as closely spaced as ˜8 μm can be found in thedataset, while the most frequent value for nearest-neighbor neurondistances is in the range of 12-16 μm.

The temporal signals corresponding to 807 active neurons identified in a30-minute example recording. It was found that the typical shapes ofCa²⁺ transients, as observed by other methods, to be reproducedfaithfully, even for the neurons at the greatest recorded depths of ˜360μm. To validate this qualitative observation and to benchmark theability of MiniLFM in combination with SID to detect and demix theactivity of nearby neurons within the scattering mammalian brain,modifications were made to the MiniLFM that allowed simultaneousfunctional ground truth information on the activity of the same neuronsto be obtained: By coupling the MiniLFM with a tabletop two-photonscanning microscope (2PM), hippocampal CA1 neurons could be excited andthe neuronal activities could be detected simultaneously through thedetection arm of the 2PM and the unmodified MiniLFM sensor module. Astate-of-the-art signal extraction algorithm_(31A) followed by humaninspection was used to establish the ground truth neuron positions andactivity traces from the 2PM data. SID-extracted positions andactivities were subsequently compared to the ground truth.

Despite the greatly reduced signal-to-noise ratios in both detectionchannels, due to the splitting of the fluorescence light into the twodetection channels, as well as coupling inefficiencies, good agreementbetween MiniLFM/SID data and the ground truth was demonstrated. It wasfound that active neurons are detected accurately (precision score:0.97±0.02) and reliably (sensitivity score: 0.79±0.04) by SID, resultingin an overall detection performance as quantified by the F-score of0.87±0.03 (mean±s.e., pooled across all recordings). More detailedexamination of the data revealed that both the locations and neuronalsignals overlap well between MiniLFM/SID and ground truth recordings. Toobtain an upper bound (conservative estimate) for the performance of SIDunder imaging conditions, the fidelity of the SID-extracted activitytraces were characterized in two ways: First, the cross-correlationbetween the individual SID-extracted traces and their ground-truthcounterparts were calculated and a median value of 0.88 was found,indicating a high general overlap. Note that in the utilized hybrid(2PM-MiniLFM) detection modality, both the obtainable signal similarity,as measured by cross-correlation, and the neuron detection performance(F-score) are limited by the achievable signal-to-noise ratio given bythe suboptimal arrangement of 2P excitation through the GRIN lens in thehybrid setup, as well as the high MiniLFM sensor gain required to detectthe signal. Under regular MiniLFM operating conditions, in which thefluorescence is generated via one-photon excitation, the signal level isorders of magnitude higher, which is expected to translate to comparableor better performance parameters during actual experiments with theMiniLFM.

Second, a metric was derived that quantifies any crosstalk thatoriginates from suboptimal demixing of neuronal activity for distinctneuronal pairs and was investigated as a function of neuronal pairdistance. To do so, the mutual information value found for each possiblepair of ground truth traces was subtracted from those of thecorresponding SID traces, and this difference was binned (“excess mutualinformation”) as a function of the distance between of the two neurons.For large neuron distances, where the effects of crosstalk arenegligible, it was observed, as expected, that the resulting excessmutual information value reaches a plateau around a low, noise-limitedbaseline. For short neuronal pair distances, however, the metric isexpected to pick up any crosstalk-induced false similarities betweentraces that would result in an unphysiological increase of the excessmutual information value. However, no such increase could be detected inthe recordings for shorter neuronal pair distances. Only when cuttingthe data to the level of individual calcium transients, eliminating thebaselines, and thereby artificially boosting the sensitivity, could aminimal but significant increase in the value of the crosstalk metric bedetected for neuronal pairs separated by less than ˜15 μm. Theseanalyses demonstrate that the approach can faithfully discriminate andachieve crosstalk-free demixing of neurons at separations around orlarger than ˜15 μm and establishes the value for what referred to as the“neuron discrimination performance.”

Contamination of neural signals by neuropil activity could be anotherconcern in a number of calcium imaging modalities, including those withreduced spatial resolution. This issue can be addressed on the molecularlevel by the using Ca²⁺ indicators with expression localized to the cellnucleus. While the localization of GCaMP expression to the nucleus canreduce the sensitivity of the response and result in slower responsetimes, it is an effective strategy to eliminate the neuropil signal.Using animals expressing a nucleus-localized version of GCaMP6,similarly well-separated sources, low or no apparent signal cross-talk,and good signal-to-noise ratio were found (despite somewhat lowerobservable overall neuronal activity). These observations, together withthe ground truth recordings and analysis suggest that neuropilcontamination is not a critical issue under the experimental conditions.While exhibiting slower dynamics, nuclearly confined indicatorseliminate crosstalk and background from neuropil and can thus beanticipated to maximize signal quality and neuron separability underconditions with extremely high densities of active neurons, a high ratioof imaging volume occupied by processes, or more severe scattering, andultimately extend the reach of MiniLFM/SID imaging to greater depths.

Minimizing motion-induced recording artifacts is essential infree-behavioral settings in which the brain and skull are naturallyexposed to a larger degree of movement. The Miniscope body andskull-attached baseplate are designed to minimize motion of the opticalsystem relative to the brain volume being imaged. Consistent with whathas been reported in the literature_(23,28), it has been found thatmotion effects are dominated by temporary lateral displacements of theFOV, an effect which is attributed to the axial rigidity of the mainbody. To minimize these displacements, in the disclosed subject matter,a baseplate has been glued rigidly to the skull over a large contactsurface, and the MiniLFM main body is attached to the baseplate usingmagnets and fixed by tightening a screw against a metal-enforced facetof the body. The absence of any moving optomechanical parts and therelatively high frame rate significantly reduce the overallsusceptibility to motion-induced perturbations of the Ca²⁺ activityreadout. The magnitude of motion-induced displacement of the recordedimage was quantified by plotting the observable lateral(anterior-posterior and lateral-medial) shifts during a 10-minuteregular (non-LFM) Miniscope recording, in which shifts are more directlyobservable than in MiniLFM/SID. The short-term lateral shifts were foundto be typically on the scale of tenths of a neuron diameter in thelateral-medial direction, and less than a neuron radius in theanterior-posterior direction. The long-term drift throughout the entirerecording is on the order of a tenth of a neuron diameter, and under theconditions is sufficiently small to allow for reliable re-identificationof neurons across days and weeks, consistent with previousobservations₂₈. Further characterized was how strong mechanical impact,as induced when the microscope on an animal's head contacts the walls ofthe arena, may lead to residual motion artefacts. To address this issue,an algorithm was developed that automatically corrects for such motionevents using a custom signal extraction pipeline that detects motionbursts in the raw imaging data, i.e. without requiring additional motionsensors. It applies the SID algorithm individually to the low-motionsegments between the bursts and then pools all neuron detections acrosssegments, exploiting the fact that neurons reliably return to theiroriginal locations in the field of view (FOV) after a motion burst asexperimentally confirmed. Finally, a model of the GCaMP responsekernel_(31A) is optimized for each neuron and subsequently used tointerpolate the activity traces across motion-affected frames. At thesame time, this model also yields a maximum-likelihood estimate of theunderlying firing rates.

The motion detection metric that underlies this approach was verified bycomparing it to data recorded simultaneously by an accelerometerattached to the MiniLFM. It was found that while not necessarily allacceleration peaks lead to motion artefacts in the functional imagingdata, the two metrics are in clear qualitative agreement.

The disclosed embodiments (MiniLFM design) thus combines LFM, SID andMiniscope technology to provide a powerful strategy that enables fastvolumetric imaging at low photobleaching and phototoxicity in scatteringtissue of freely moving animals. The MiniLFM design establishes a simpleand extensible platform that can be easily customized and adapted toother model animals. Together with the computational efficiency andneuron discrimination capability of the SID algorithm, the approach thusoffers a unique platform for population-level studies of neuralinformation processing in freely behaving animals and allows theanalysis of the neuronal basis of social interaction.

Methods of Extracting Signal Information

In one embodiment, the operations performed to demix signal informationinclude the following as discussed herein. A 2D standard deviation imageis generated from information obtained from the imaging apparatusinterface. The 2D standard deviation image estimates the ballisticcomponent of the imaging information. Next, a 3D image is generated byremapping (e.g., deconvolving) the 2D standard deviation image. From the3D image, a candidate object is identified. Next, an estimated spatialforward model of the candidate object is obtained by mapping (e.g.,convolving) the 3D image of the candidate object with a PSF associatedwith the imaging apparatus. Next, background-corrected data is obtainedby using the estimated spatial forward model of the candidate object andestimated temporal components. The estimated spatial forward model andestimated temporal components are iteratively updated until convergenceis reached for the candidate object, thereby demixing the signalinformation.

In one embodiment, before the 2D standard deviation image is generated,background information obtained by the imaging apparatus is subtractedusing the imaging apparatus interface. In one embodiment, the backgroundinformation is background fluorescence obtained from the LFM. In oneembodiment, subtraction of the background information includes applyingrank-1-matrix factorization.

In one embodiment, the 2D standard deviation image is generated byestimating the ballistic component of the emitted signal by taking thestandard deviation of the time series of camera frames. Since ballisticphotons are spread across fewer sensor pixels than scattered light,signals from ballistically illuminated pixels have a higher variation intime for a given underlying source activity, and thus can be separatedfrom the scattered component.

In one embodiment, the 3D image generated by remapping (e.g.,deconvolving) the 2D standard deviation image includes unraveling 3Dposition information from the 2D image (e.g., 2D standard deviationimage) by remapping (e.g., deconvolving) the 2D image with thenumerically simulated, ballistic PSF of the associated imagingapparatus. In the presence of scattering, this approach results involumes containing vastly sharper sources and reduced background thanwhat would be obtained by deconvolving the raw data directly andsubsequently calculating the standard deviation of the result. In oneembodiment, before the 3D image is generated, the 2D image isthresholded to exclude residual background activity. In one embodiment,generation of the 3D image further includes reducing reconstructionartefacts by incorporating total-variation and sparsity constraints intothe deconvolution. For example, reducing reconstruction artefacts caninclude applying the following equation:x _(n+1) =x _(n)(P ^(T) y/P y+λ1_(dim(x))),  (1)wherein x represents a volume estimate, 1_(dim(x)) represents a vectorof ones with the same dimension as x, P represents thepoint-spread-function, λ represents weight of a sparsity-encouragingterm, and y represents background subtracted raw data.

A candidate object can be any spatially confined signal-emitting entity.In one embodiment, identification of a candidate object includes usingspatial segmentation to suppress spatial frequencies incompatible withobject shapes. Examples of object shapes can be any part of the anatomyof a biological being, including for example, a neuron, organ, bone,muscle, cellular structure, and/or tumorous growth. For example, neuronscan be localized and separated in the 3D image, i.e., the reconstructed3D volume. In one embodiment, the spatial segmentation includes applyinga bandpass filter to the 3D image, thresholding to exclude backgroundartefacts, and applying a local maximum search algorithm. Thesegmentation threshold is chosen to robustly reject noise and artefacts.

In one embodiment, the estimated spatial forward model of the candidateobject obtained by mapping (e.g., convolving) the 3D image of thecandidate object with a PSF includes producing a sparse non-negative p×nmatrix S_(i), wherein n is the number of object candidates, p is thenumber of pixels, i is the iteration number, and S₀ is the initialspatial forward model of the candidate object. For example, for eachidentified candidate object, the expected LFM footprint (e.g., itsexpected camera sensor pattern) is calculated by mapping (e.g.,convolving) the 3D image of the candidate object with the PSF associatedwith the imaging apparatus.

In one embodiment, the background-corrected data obtained by using theestimated spatial forward model of the candidate object and estimatedtemporal components includes generating a p×t matrix Y using the matrixproduct of S₀ and T₀, wherein T₁ is a non-negative n×t matrix oftemporal components, and t is the number of time steps in the recording.In one embodiment, T_(i) is obtained by iteratively applying an adaptedRichardson-Lucy-type solver with a sparsity constraint.

In one embodiment, iteratively updating the estimated spatial forwardmodel and estimated temporal components includes i) obtaining an updatedestimate of S_(i) while keeping estimated T_(i) constant, obtaining anupdated estimate of T_(i) while keeping estimated S_(i) constant, andii) iteratively repeating operation (i) until convergence is reached,for the object candidate. For example, an updated forward model estimateS₁ is found while keeping T₀ constant. In one embodiment, the problem isbroken down by grouping the signals corresponding to spatiallyoverlapping sets of components into k smaller matrices T₀ ^(k) andfinding updated spatial component estimates S₁ ^(k) by solving anon-negative least-squares problem. During this update step, the rows ofS₁ ^(k) are forced to be zero outside of pre-defined masks derived fromthe ballistic footprints to ensure compact solutions. This procedure isiterated until convergence. Such procedure is a bi-convex optimizationproblem solved by alternatingly iterating the temporal and spatialupdate operations until convergence is reached.

In one embodiment, an iterative source extraction procedure forscattered LFM data, which is referred to as SID is provided. Thisprocedure achieves accurate neuron localization and signal demixing byseeding inference with information obtained from remnant ballisticlight. The estimates of the time series and the scattered images of eachactive neuron are iteratively updated by non-negative, constrainedleast-squares optimization.

The disclosed embodiment of SID represents a new scalable approach forrecording volumetric neuronal population activity at high speed anddepth in scattering tissue. This was done by addressing two keylimitations of LFM for Ca²⁺ imaging: the lack of robustness toscattering and high computational cost. The disclosed embodiments allowextending the application of LFM beyond semi-transparent model organismsto the scattering mammalian brain, enabling large-FOV, high volume ratereadout of neuronal activity across multiple cortical layers in awakerodents. Such embodiments enable to reliably extract neuronal activitytraces of cells expressing genetically encoded Ca²⁺ indicators within avolume of ˜900×900×260 μm in the mouse cortex, located as deep as 380 μmand at 30 Hz volume rate at a discriminability performance of 20 μm, aswell as from similarly sized volumes in the mouse hippocampus.

Seeding the SID demixing algorithm with an initial estimate of sourcelocation information enables recovery of dynamical information fromscattered photons in recordings, consistent with what is expected basedon the scattering and isotropy parameters of the brain tissue. Thedisclosed embodiments highlight the advance of combining optical imagingwith jointly designed computational algorithms to extract informationfrom scattering media.

SID can robustly detect neurons at least to a depth of ˜375 μm andrecover the majority of actual neuronal signals with high fidelity inthe presence of active neuropil. Compared to other existing methods forhigh-speed volumetric Ca²⁺ imaging_(9,15,17-22), SID stands out by itscombined acquisition volume and speed, its simplicity and exceptionallylow cost as well as its extreme scalability.

While some sequential acquisition methods based on 2P excitation mayprovide higher spatial resolution, unlike these, the voxel acquisitionrate and resolution in SID are independent of the size of the acquiredsample volume and only limited by the camera frame rate (up to 100 Hz)and fluorophore properties. It is, therefore, conceivable to extend SIDto much larger FOVs without sacrificing its performance in speed andresolution, while at some point the combined obtainable volume size andspeed in 2P techniques will be ultimately limited by tissue heating.

In contrast to single-photon techniques_(5,26,27) including the variousimplementations of light sheet microscopy, SID extracts information fromthe scattered light allowing it to image in scattering specimen beyondwhat has been shown for other single photon techniques.

In one embodiment, the depth penetration, which may be affected bybackground fluorescence emerging from below the reconstructed volume, isaddressed. In this embodiment, PSFs are modeled with a larger axialrange which would be able to explain more of the recorded light in termsof localized sources rather than in terms of a diffuse background.Labelled and active neuropil contribute to this background, and hencesoma-confined or nucleus-restricted Ca²⁺ reporters assist to increasethe obtainable depth range and the quality of the extracted signals.

In one embodiment, there is a correction for wavefront distortionscaused by tissue inhomogeneities using adaptive optics 48 to increaseresolution and source separability. Many biological applications may notrequire high labelling density, but rather targeted or sparse labeling,thus reducing background and greatly easing the task of neuronal signalassignment and demixing. Furthermore, GECIs fluorescing at longerwavelengths are generally beneficial for deep-tissue imaging, due to theincreased scattering length in the red and near-infrared region of thespectrum.

Faithful extraction of neuronal signals may be limited by the loss ofdirectional information due to multiple photon scattering. The criticaldepth for information loss is known as the transport mean free path anddepends on the scattering length anisotropy parameter. In the mousebrain, it amounts to ˜10 scattering lengths, or 500-1000 μm₇.

Previous implementations of image reconstruction and data extraction inLFM microscopy typically involved the use of a computing cluster₄, whichseverely limits both its dissemination among biological users and itsuse in real-time and closed loop applications. The disclosed SID rendersthis problem tractable on an individual workstation, enabling volumetricreadout across multiple cortical areas and layers at unprecedented speedusing widely available, simple hardware. In this context, the disclosedembodiments demonstrated three-order-of magnitude reduction incomputational burden is not merely an incremental improvement but rathera transformative step that allows LFM-derived volumetric imagingapproaches far exceeding existing scale and versatility. Computationalimaging, especially plenoptic recording technologies such as LFM,combined with advanced machine learning for neuron identification andsignal extraction₄₇ vastly improve the reach, applicability andacuteness of optical sensing.

Examples

The following examples confirm the effectiveness of the disclosedapproaches using simulated data sets. In comparison to conventionaldeconvolution, the disclosed embodiments provide robust signal demixingup to a depth of about four scattering lengths (corresponding to up to˜400 μm in a mouse cortex). In addition, when applied to weaklyscattering samples such as larval zebrafish, the disclosed algorithmdelivers increased temporal and spatial fidelity.

To verify and characterize the demixing performance of the SID approach,it was applied to synthetic datasets containing randomly positionedneurons with partially correlated, GECI-like activity. A simulatedscattered PSF using a Monte-Carlo approach₃₈ was generated, using valuesfrom literature for its parameters_(7,39). Then, volumetric framescontaining the randomly positioned neurons with the scattered PSF wereconvolved to yield synthetic LFM raw data corresponding to a depth ofapprox. 400 μm in mouse cortex. Camera noise and background fluorescencewas added with signal-to-background and signal-to-noise ratios chosen tomatch experimental data. Application of the SID algorithm to thesynthesized data reliably demixed overlapping spatial footprints, and incases where naïve signal extraction would give highly mixed signals, SIDallowed for faithful signal demixing yielding close correspondence (meancorrelation of 0.76) of the extracted signals. SID was found to requireonly a small difference in temporal activity and spatial footprint tofaithfully differentiate the two entities.

Seeded Iterative Demixing (SID) Improves Source Localization inZebrafish Larvae

LFM-based Ca₂₊ imaging has been shown to be capable of capturingneuronal activity from large parts of the brains of zebrafish larvae.While the unpigmented mutants commonly used for these experiments haveremarkably low light absorption, these mutants are not fully transparentand exhibit some amount of scattering. Zebrafish larvae are therefore anideal testbed for the present enhanced source extraction method. Whileallowing a baseline performance in the weak scattering regime to beestablished, imaging the larval zebrafish brain poses the additionaldifficulty of a higher neuron density than in the mammalian cortex.

In LFM, the lateral resolution is traded off with the ability to collectangular information from the light field. The parameters of the LFMdesign were chosen to yield a lateral resolution of 3.5 μm,corresponding to about half a neuron diameter in zebrafish larvae 5, anda field-of-view FOV of 700×700×200 μm, which is large enough to capturethe brain from the olfactory bulb to the anterior part of the hindbrain.

Employing a custom hybrid two-photon and light-field microscope, theneuron positions extracted via SID were compared to a high-resolution2PM image stack, using a volume of 775×195×200 μm in the anterior partof the zebrafish spinal cord. Spatial segmentation of the 2PM stackyielded a total of 1337 neurons within the above volume, which includesboth active and inactive neurons. SID inherently detects active neuronsonly, and yielded 508 neurons whose positions clearly coincide withneurons in the 2PM stack. Spontaneous neuronal activity from the entirelarval zebrafish brain covering a volume of 700×700×200 μm at 20 fps forfour minutes was recorded. In this case SID found a total of 5505 activeneurons.

Signals and neuron locations identified by SID were compared with anICA-based analysis after conventional reconstruction of the same data.While in many cases ICA and SID yield matching pairs of positions andsignals, it was found that ICA tends to over-segment the data bysplitting up a neuron into several spatial filters with largely similarsignals. Moreover, ICA-based analysis is also prone to identifying areasthat contain scattered contributions from several surrounding neurons asfalse positive neurons, resulting in duplicate signals that exhibitsevere crosstalk.

Overall, it was found that, when compared with ICA, SID typicallyidentifies considerably more (˜50% in this example) of the activeneurons. Furthermore, the majority of signals identified by ICA werealso recovered by SID (>0.8 cross-correlation between ICA and SID for82% of ICA signals in the full image volume). At the same time, SIDreliably rejects false positive signals identified by ICA.

Seeded Iterative Demixing (SID) Enables High-Speed Volumetric Ca²⁺Imaging in Mouse Cortex and Hippocampus at 380 μm Depth

The severity of degradation due to scattering in standard LFMreconstruction becomes strikingly apparent when in vivo LFM data fromthe mouse cortex is conventionally reconstructed. When applying SID toLFM recordings acquired at various depths in the posterior parietalcortex of awake mice, the effectiveness of the disclosed embodimentsbecame clear. The activity of neurons expressing GCaMP6m within a volumewith a lateral FOV of ˜900 μm diameter up to a depth of 380 μm at avolume acquisition rate of 30 fps was recorded using a cranial window.The computational efficiency of this approach enables reliableassignment of neuron positions and activity traces over larger axialranges while at the same time greatly reducing computational cost. Thisallowed capture of locations and activities of neurons in mouse corticallayers I-III and part of the layer IV at 30 fps volume rate with onlytwo successive recordings. The disclosed algorithm identified over 500active neurons during a one-minute recording, corresponding to ˜10% ofall labeled neurons (5296) identified using a high-resolution 2PM. Ofthe total number of active neurons, 296 were in a depth range from zeroto 170 μm, and 208 active neurons in a range from 120 to 380 μm.

The disclosed algorithm allows for some tradeoff between false positivesignals versus sensitivity to weak signals that can be adjusted by theuser based on the biological question being studied. For all the resultsdiscussed herein, a rather conservative extraction strategy was usedthat prioritizes rejection of false positives over sensitivity to weaksignals. Such a setting, along with the enforcement of post-selectionbased on spatial shape also allows for a more efficient rejection of theneuropil signal. However, depending on the biological question and GECIproperties, the extraction strategy can also be tuned to result in lessconservative estimates.

To further illustrate the versatility of SID, the disclosed method wasapplied to imaging of CA1 hippocampal neurons using a cranial windowimplanted after cortical aspiration_(40,41). Capturing the neuronalpopulation activity within a volume of ˜900×900×200 μm containing thecell body layer of CA1 neurons, Ca²⁺ signals from 150 neurons arrangedin the curved layer geometry typical of the anatomy of this region couldbe reliably identified, extracted, and demixed. The robust andpronounced Ca²⁺ transients extracted by SID are consistent with thehigh-frequency bursts of neuron types in this brain region₄₂. Insummary, it was shown that SID reveals neuron positions and temporalsignals to a depth of up to 380 μm in mouse cortex and hippocampus invivo. In the next section, the extraction fidelity of the disclosedembodiments is verified by comparing it to 2PM recordings.

Seeded Iterative Demixing (SID) Allows for Demixing of OverlappingNeuronal Signals in the Mouse Brain while Providing Time SeriesConsistent with 2PM

The capability of SID to demix neuronal signals in scattering tissuewhile providing neuronal time series that closely match those obtainedby more established methods, such as 2PM, was experimentallydemonstrated. Taking two CA1 neurons that are indistinguishable based ontheir spatial footprints, and which exhibit highly correlated activity,it was shown that SID can separate these neurons spatially and demixtheir time signals. To achieve this, SID requires only a few pixelswithin the spatial footprint of each neuron to eliminate crosstalk fromthe remaining neuron. The volumetric FOV and frame rate of LFM exceedthose of other methods, such as 2PM, that are typically used for in vivoCa²⁺ imaging at similar depths in the mouse cortex. It is, therefore,impossible to establish an experimental ground truth for the disclosedembodiments by directly comparing the neuronal time series obtained bySID and 2PM within typical LFM volume sizes and volume acquisitionrates. Nevertheless, experimental ground truth data was generated andvalidated that time series extracted using SID are indeed consistentwith data from more established methods such as 2PM, within the limitsof current technology. To do so, a 2PM excitation was performed in asingle plane in the mouse cortex while simultaneously detecting thefluorescence using an LFM detection arm and a photomultiplier tube (PMT)point detector in the hybrid 2PM-LFM. The 2PM hardware allowed scanninga plane of 200×200 μm at 5 Hz. When comparing localization and signalextraction for twelve neurons found in this region using spatialsegmentation on the obtained 2PM data, and SID on the LFM detection arm,it is clearly demonstrated that signals extracted by SID are inquantitative agreement with 2PM recordings, that yields 12 out of 12active neurons detected, and a mean cross-correlation of signals fromthe two methods of 0.85.

Seeded Iterative Demixing (SID) Allows for Demixing and Localization ofOverlapping Neuronal Signals in the Mouse Brain with Time SeriesConsistent with 2PM Ground Truth

Next, the capability of SID to demix neuronal signals in scatteringtissue while providing neuronal time series that closely match thoseobtained by more established methods, such as 2PM, was experimentallyand systematically demonstrated. As an example on the single-neuronlevel, two CA1 neurons that were indistinguishable based on theirspatial sensor footprints, and which exhibit highly correlated activity,were selected. SID can detect the neurons as individual neuronsspatially and demix their corresponding time signals. To achieve this,SID only requires a few pixels within the spatial footprint of eachneuron to eliminate crosstalk from the respective other neuron.

The volumetric FOV and frame rate of the disclosed embodiment exceedthose of other techniques such as 2PM that are typically used for invivo Ca²⁺ imaging at similar depths in the mouse cortex. It is thereforeimpossible to establish an experimental ground truth for the disclosedembodiment by directly comparing the neuronal time series obtained bySID and 2PM within the typical volume sizes and volume acquisitionrates. Nevertheless, experimental ground truth data were generated andtime series extracted by SID were validated as being consistent withdata from more established methods such as 2PM, within the limits ofcurrent technology. Such was done using a hybrid 2PM-SID microscope (seeMethods). 2PM excitation was performed in a single plane in the mousecortex while simultaneously detecting the fluorescence using the SIDdetection arm and a photomultiplier tube (PMT) point detector in thedisclosed hybrid 2PM-SID. The 2PM hardware allowed scanning of a planeof 200×200 μm at 5 Hz. When comparing localization and signal extractionfor twelve neurons found in this region using spatial segmentation basedon watershed transform on the obtained 2PM data, and SID on dataobtained in the LFM detection arm, it is clearly demonstrated thatsignals extracted by SID are in quantitative agreement with 2PMrecordings (12 out of 12 active neurons detected; mean cross-correlationof signals from the two methods: 0.85).

To obtain a more comprehensive and quantitative evaluation of SIDsperformance, a set of single-plane, simultaneous 2PM-SID movies at aseries of axial depths (100-375 μm, total n=18 recordings) wererecorded. Neuron positions and signals were extracted from the 2PMchannel using a recently published and increasingly used method₃₆ basedon constrained matrix factorization (“CaImAn”). The output of CaImAn wasassessed and corrected manually to establish a ground truth, to whichboth the raw CaImAn output and SID were quantitatively compared.

In FIG. 3a , the neuron detection performance of the two methods atdifferent tissue depths were illustrated by plotting the ratios of trueneurons that were detected correctly, the “Sensitivity” score (FIG. 3a(i), the ratio of false positive detections to total detections,“Precision” (FIG. 3a (ii), and the harmonic mean of these twoquantities, the “F-Score” (FIG. 3a (iii). While there is a tradeoffbetween Sensitivity and Precision, F-Score can be used as a parameter tocharacterize the overall performance of each method. Both methodsidentify most actual neurons correctly (FIG. 3a ). However, SID is lessprone to false positive classifications (FIG. 3b ). Overall, SID offersa comparable or better compromise between sensitivity (sensitivityscore) and robustness (Precision score) resulting in slightly higherF-Scores.

The quality of the SID-extracted neuronal activity traces compared toground truth was characterized at different depths in FIG. 3b . The meancorrelation between SID-extracted and 2PM ground truth signals decaysonly moderately from 0.84±0.05 at 100 μm depth to 0.77±0.05 at 375 μm(FIG. 3b (i)). Of all true positive SID detections, 73% have acorrelation with ground truth of better than 0.8, and 60% better than0.9 (FIG. 3b (ii) histogram and FIG. 3b (iii) example trace pairs) whileonly 10% of extracted signals exhibit a low (<0.4) correlation with 2PMground truth and correspondingly a degraded overlap of the neuronalsignal due to crosstalk with nearby neuropil. To gain an insight intothe dependence of such mismatches as a function of tissue depth, how thefraction of SID-extracted neurons with a correlation to ground truth ofless than 0.5 depended on tissue depth was calculated (FIG. 3b (iv)).Their fraction was found to represent only 6% at 100 μm depth and about12% at 375 μm. While this shows that SID can correctly identify andassign neuronal signals for the vast majority of neurons even in adensely-labeled sample, as the main source of the above mismatches wereinteractions with the neuropil. Even better results are obtained byeliminating neuropil labelling by using soma- or nucleus-confined Ca²⁺indicators. In addition, a computational strategy for demixing andrejecting neuropil contributions from the signals was also outlined.

Next, SID's performance to demix signals of nearby neurons wasinvestigated. Both physiological correlation of neuronal signals, whichare known to generally increase with decreasing neurons pairs distances,as well as degradation of SID's performance at short neuron pairdistances are expected to result in an increase in the observedcorrelation for decreasing distance of neuron pairs. To dissect theunderlying drivers of such observed correlations for the SID extractedpairs, their dependence on whether the underlying ground truth pairdynamics was correlated or uncorrelated was investigated. To identifysuch ground truth neuronal pairs, the corresponding cross-correlationmatrix and histogram were calculated. Subsequently, all uncorrelatedneuronal pairs (<0.2) as well as correlated neuronal pairs (>0.6) wereselected and the correlations of the corresponding signal pairs in SIDwere examined. An increase in correlation for pairs of uncorrelatedground truth neurons for separations smaller than ˜20 μm was found;while for pairs with correlated ground truth activity, the correspondingSID extracted pairs exhibited a similar correlation as their groundtruth pairs over a range of lateral distances and for as close as ˜20μm. The above un-physiological increase in the observed correlation foruncorrelated ground truth neuron pairs extracted by SID below ˜20 μm, aswell as the consistency of SID with correlated ground truth pairs, downto approximately the same distance, provides a metric that representsthe discriminability achieved by the disclosed SID algorithm, i.e. itsability to detect and assign neuronal time series in the scatteringmouse brain. The limit of SID is reached when SID starts to detectartificial “correlations” between neurons known to be un-correlated.

Methods

Hybrid Light Field and Two-Photon Microscope

The microscope used for simultaneous 2PM and LFM imaging, the fishrecordings and mouse recordings is built around a Scientifica Slicescopeplatform with a custom LFM detection arm.

The two-photon excitation source (Coherent Chameleon) delivered 140 fspulses at 80 MHz repetition rate and 920 nm wavelength. The beamintensity was controlled via an electro-optical modulator (Conoptics)for attenuation and blanking, and fed into a galvo-based scan head(Scientifica). The 2P path and the one-photon excitation/LFM detectionpath were combined via a short-pass dichroic mirror (SemrockFF746-SDi01). One-photon excitation light from a blue LED (CoolLED pe-2)was fed into an Olympus epi-fluorescence illuminator and reflected intothe LFM detection path via a standard EGFP excitation filter anddichroic.

Depending on the experiment, either one-photon or two-photon light wasused while the other was blocked. Either was focused by a Nikon 16×0.8NAwater-dipping physiology objective into the sample. For zebrafishexperiments, Olympus 20×1.0NA and Olympus 20×0.5NA water-dippingobjectives were used.

Fluorescence from the sample was detected either by a non-descanned PMTarm, or the LFM arm, or split among both. The split ratio was determinedby a main beam splitter inserted into the beam path behind theobjective. A custom detection head design allowed for quick switchingbetween configurations that route 100% to the PMTs (665 nm long-passdichroic, Scientifica), 100% to the LFM arm (no filter), or split thefluorescence 10:90 or 50:50 (PMT:LFM) (Omega 10% beam sampler orThorlabs 50:50 vis beam splitter, respectively). The PMT detection armconsisted of an IR blocking filter, collection lens, 565LP dichroic, and525/50 nm and 620/60 nm emission filters with, and Scientifica GaAsP(green channel) and alkali (red) PMT modules.

For LFM detection, fluorescence passed through the short-pass dichroicthat couples the laser into the beam path, as well as the one-photonfilter cube. The image formed by a standard Olympus tube lens was thenrelayed via two 2-inch achromatic lenses (f=200 mm, Thorlabs) onto amicrolens array (MLA, Okotech, custom model, size 1″ square, f-number10, 114 μm microlens pitch, quadratic grid, no gaps). The f-number ofthe MLA was matched to the output f-number of the microscope. The backfocal plane of the MLA was relayed by a photography macro objective(Nikon 105 mm/2.8) at unity magnification onto the sensor of an AndorZyla 5.5 sCMOS scientific camera, which can be read out at up to 75 fpsat full resolution (2560×2160 px, 16 bit).

The setup was controlled from a dual-CPU workstation (HP Z820) with foursolid-state disks in a RAID-0 configuration for fast image acquisitionand National Instruments 6110 and 6321 cards for analogue and timingI/O. Experiments were controlled using Micro-manager and Scanimage forthe one-photon and two-photon parts of the setup, respectively.

Source Extraction Algorithm and Data Analysis

The disclosed source extraction approach starts with a rank-1 matrixfactorization of the time series of raw images to remove background andcommon-mode dynamics. A motion detection metric is computed on thebackground-subtracted images, and frames with a motion metric valueabove threshold are excluded from further processing. Next, the standarddeviation of each pixel along time is computed, resulting in a “standarddeviation image.” The standard deviation image was deconvolved using aRichardson-Lucy-type algorithm (with non-negativity and, optionally,sparsity constraints) and a numerically simulated PSF, as describedpreviously 4,29. This results in a volumetric frame containing neuronsthat are active in the recording as bright regions. The reconstructedvolume is band-pass filtered and segmented using a local maximum search,resulting in a dictionary of neuron candidate positions. Each positionis convolved with the simulated PSF to obtain an initial estimate of its(ballistic) footprint on the LFM camera. From each footprint, a Booleanmask mi was generated that is one at every pixel behind every microlensthat receives a contribution from the ballistic footprint. The set ofneuron footprints was collected into a non-negative p×n matrix S0, withn being the number of neurons found in the segmentation, and p thenumber of camera pixels. Also, let Y be the p×t non-negative data matrix(with t the number of time steps in the recording). A temporal updatestep is then performed by solving the non-negative least squaresproblem:minimize T∥Y−ST∥2subject to T≥0,where T is a non-negative n×t matrix T of temporal components, using aniterative solver. The background components found in the rank-1 matrixfactorization performed earlier are inserted as an additional row andcolumn of the S and T matrices, respectively, and therefore updatedtogether with the neuron candidates.

Next, a spatial update step is performed: All sets O^(k) of spatiallyoverlapping components are found. For each of these k groups, matricesT^(k) are formed, which contain all columns t, of T that correspond tospatial components in O^(k), and data matrices Y^(k) that contain onlythose pixels that fall into the nonzero areas of masks mi in O^(k). Foreach k, solve the following non-negative, spatially constrainedleast-squares problem is solved:minimize S ^(k) ∥Y ^(k) −S ^(k) T ^(k)∥2subject to S ^(k)≥0,rows of S ^(k)=0 where masks mi=0(∀i∈O ^(k)).Then, the temporal and spatial update steps are iterated untilconvergence.

Finally, the integral of every spatial component is computed, which isnormalized to one, and the temporal component are scaled by theintegral. The temporal components are scaled individually to thestandard deviation of the noise they contain (defined as the residual ofa Savitzky-Golay fit).

Signal Extraction from Frame-by-Frame-Reconstructed LFM Datasets.

In order to extract the signals and spatial filters from standard LFMdatasets (i.e., series of volumetric frames obtained by deconvolving theraw frames individually using a Richardson-Lucy type algorithm and anumerically simulated PSF), a custom Matlab implementation of anapproach based on Ref. 35 was used: After fitting and dividing out aslowly-varying trend function from the data, the variances of all voxelsover time were computed and the voxels above the 80th percentile of thevariance distribution were selected to reduce problem size. PrincipalComponent Analysis (PCA) is performed on the selected voxel time series.In order to avoid overfitting and to de-noise the data, the first 8% ofPCA components are kept and fed into the FastICA Matlab package. Theresulting ICA spatial components are post-selected based on their weightdistribution: Only those containing prominent peaks (i.e., regions withvalues larger than the 20th percentile of the weight distribution) thatare compatible in shape with a neuron are kept. The correspondingsignals are extracted from the de-trended data by averaging over allvoxels in the peak.

In-Vivo Ca²⁺ Imaging of Head-Fixed Zebrafish Larvae.

For zebrafish experiments, elav13:H2B-GCaMP6s fish (n=4) were imaged 5-8days post fertilization. This line expresses a nuclear confined calciumindicator pan-neuronally in a mitfa−/−, roy−/− background. Larvae wereimmobilized by embedding them in 2% low melting point agarose. Forspinal cord recordings, larvae were paralyzed by injection ofα-bungarotoxin (125 μM) into the heart cavity at least one hour beforethe experiment.

Animal Surgery and In-Vivo Ca²⁺ Imaging of Awake Mice.

Surgery and experimental procedures fulfilled the Austrian and Europeanregulations for animal experiments (Austrian § 26 Tierversuchsgesetz2012—TVG 2012) and were approved by the IACUC of The RockefellerUniversity. Adult (P90+) male and female C57Bl/6J wild-type mice (n=10)were anesthetized with isoflurane (2-3% flow rate of 0.5-0.7 l/min) andplaced in a stereotaxic frame (RWD Life Science Co., Ltd. China). Afterremoving the scalp and clearing the skull of connective tissues, acustom-made lightweight metal head-bar was fixed onto the skull withcyanoacrylate adhesive (Krazy Glue) and covered with black dental cement(Ortho-Jet, Lang Dental, USA or Paladur, Heraeus Kulzer, GmbH, Germany).The head-bar was stabilized by anchoring it with up to 3 headless M1.4screws inserted at the occipital and parietal bones. A circularcraniotomy (3-5 mm diameter) was then performed above the imaging site(posterior parietal cortex, PPC, centered at ˜2.5 mm caudal and ˜1.8 mmlateral; primary motor cortex, M1, −2.5 mm anterior and 1.5 mm lateral;dorsal hippocampus 2.0-2.5 mm caudal and 1.4-1.8 mm lateral to bregma).With the skull opened and the dura intact, the GECI-carrying virusAAV8:hSyn-GCaMP6m was injected at 4-12 sites (25 nl each, at 10 nl/min;titer ˜10¹² viral particles/ml) with a 400 μm spacing forming a gridnear the center of the craniotomy, at a depth of 400-450 μm below durafor PPC and 1200 μm for hippocampus. The construct AAV2/1: Hsyn-JRGECOwas injected. After the injections, a glass cranial window consisting ofa 3-5 mm diameter, #1 thickness (0.16 mm) coverslip was implanted in thecraniotomy, flushed with saline solution, placed in contact with thebrain surface, and sealed in place using tissue adhesive (Vetbond). Theexposed skull surrounding the cranial window was covered with dentalcement to build a small chamber for imaging with a water-immersionobjective. To access the dorsal hippocampus, a cranial window wasimplanted after cortical aspiration as previously reported. 42,43 Toprevent post-surgical infections and post-surgical pain, the animalswere supplied with water containing the antibiotic enrofloxacin (50mg/Kg) and the pain killer carprofen (5 mg/Kg) for a period of ˜7 days.After surgery, animals were returned to their home cages for 2-3 weeksfor recovery and viral gene expression before subjecting to imagingexperiments. Extreme care was taken to ensure that the dura experiencedno damage or major bleeding before and after cranial windowimplantation. Mice with damaged dura or unclear windows were euthanizedand not used for imaging experiments. During imaging sessions, theanimals were head-fixed using a customized mount complemented with ahead bar holder and a mouse body stabilizer (body jacket) and couldfreely run on a disk (200 mm diameter). Spontaneous activity wasrecorded. This considerably reduced animal induced motion of the brainduring imaging. A ventilation mask was placed in front of the mouse noseto provide air puff mechanical stimulation to the mouse whiskers andface as well as to provide gas anesthesia on demand. Typical imagingsession lasted continuously for 2-10 min.

SID Algorithm Implementation Details

Background Rejection

Deep tissue LFM movies contain strong global background fluorescencewhich has to be subtracted before computing a standard deviation imageand before any further steps. This background is mostly due tofluorescence originating from above and below the depth range capturedby the numerically simulated PSF that is used for reconstruction. Thisbackground was extracted by applying a rank-1-matrix-factorization tothe LFM raw data. The spatial and temporal components obtained fromrank-1-matrix-factorization are added to the neuron candidates in thespatial and temporal update steps as an additional row and column of theS and T matrices, respectively. The background estimates are thereforerefined during these optimization steps, and activity may bere-allocated from neurons to the background, and vice versa. In thetemporal update step, this corresponds to an inherent backgroundsubtraction, while in the spatial update step, the shape of thebackground is refined.

Without background subtraction, the standard deviation image of an LFMmovie is dominated by temporal variations in the background. Aone-dimensional approximation of the background was sufficient to obtainthe ballistic components of the neuron footprints. The standarddeviation image was compared without and with background subtraction,respectively. It is evident that removing the background reveals LFMfootprints of localized sources.

Reconstruction with Sparsity, Segmentation

The standard deviation images were reconstructed (de-convolved withnumerically simulated PSF) using a modification of aRichardson-Lucy-type algorithm known as ISRA1, which yields non-negativecomponents. Classical LFM reconstruction based on Richardson-Lucydeconvolution with a ballistic PSF_(2,3) is prone to blocky artefactsnear the native focal plane of the microscope where the optical spatialsampling density is strongly reduced.₂ These artefacts are detrimentalto the success of the subsequent segmentation procedure. When necessary,ISRA was modified with a sparsity constraint. The update step for volumeestimate x is:x _(n+1) =x _(n)(P ^(T) y/P ^(T) P _(y+λ)1_(dim(x)))where 1_(dim(x)) is a vector of ones with the same dimension as x, and Pis the PSF. The parameter λ governs the weight of thesparsity-encouraging term. λ>0 was used for the zebrafish recordings.For deep mouse recordings, λ=0 was set for performance reasons andinstead discarded neuron candidates detected in the artefact region.Before reconstruction, standard deviation images were thresholded toexclude residual background activity.Segmentation

In order to suppress spatial frequencies not compatible with neuronshapes, a bandpass filter was applied to the reconstructed standarddeviation volume, followed by thresholding the result to excludebackground. Then, a local maximum search algorithm was applied. Detectedregions in a reconstructed standard deviation image are labelled withred dots. The segmentation threshold is chosen to robustly reject noiseand artefacts.

Non-Negative Matrix Factorization

The algorithm proceeds as described in the Methods section of the maintext, by alternating temporal and spatial update steps. While theinitial spatial estimate only includes the ballistic footprint, theupdated estimate increasingly incorporates the scattered light aroundit. The corresponding temporal components become more pronounced andincreasingly de-mixed from overlapping signals.

Convergence

Both the spatial and temporal optimization steps are convex problemsand, therefore, each converge to a global optimum. The combined problemis bi-convex and a variant of what is known as an alternate convexsearch₄ in the literature, which is a frequently used algorithm for thisclass of problem. The alternate convex search algorithm optimizes abi-convex target function by splitting the problem into its convexsub-problems, initializes the solution with a guess, and iterativelysolves one of the two sub-problems, while keeping the other variablefixed at the optimum of the previously solved sub-problem (or theinitial guess), and then alternating the sub-problems until a stoppingcriterion is reached. It has been shown 4 that the iteration sequencepursued by the alternate convex search algorithm has at least oneaccumulation point, and that if each accumulation point has a uniquesolution for each of the sub-problems, then the difference betweenconsecutive iterations converges to zero. The value of the targetfunction is the same at each accumulation point, and reaches a partialoptimum (i.e., an optimum in each of the convex variables). In a strictsense, the global optimality of the solution is not guaranteed. However,alternate convex search is routinely applied to bi-convex optimizationproblems, for instance in the context of Ca²⁺ imaging forspatio-temporal demixing of 2PM data₅, with good success.

For both the spatial and temporal update steps, the ISRA algorithm wasused without a sparsity constraint. It was found to parallelizeefficiently across multiple CPU-cores as well as thousands of GPU-cores,allowing for quick solution of large problems (thousands of pixels timesthousands of time steps within approximately 1 GPU-second per neuron).Fast convergence and aborting the algorithm after approximately 10iterations was routinely observed, when the residual has been reduced byfour orders of magnitude. At such point, no spatial or temporalstructure is evident in the residual data.

Synthetic Dataset Generation

The synthetic dataset was generated as follows, using literature valuesfor the parameters₇₋₉: 40 neurons (spheres of 8 μm diameter) wererandomly placed in a volume of 70×70×200 μm, maintaining a minimumdistance of one neuron diameter, and surrounded by a margin of 25 μm oneach side to avoid border artefacts. The simulated neuron density waschosen to be 40,000 per cubic millimeter. This is lower by a factor ofapproximately two than the average density reported for mouse cortex₁₀,to account for the fact that not all neurons are active during a givenrecording. The volume size was chosen large enough to span most of theLFM axial range, and for scattered neuron images originating fromdistant sides of the volume to be non-overlapping on the simulated LFMsensor, while keeping computational effort within the capacity of a20-CPU-core, quad-GPU workstation. Poissonian spike trains of actionpotentials were randomly generated (mean firing rate 0.5 Hz, 1000 timesteps at a 5 Hz sampling rate), linearly mixed to introduce somecorrelation among them (mixing matrix chosen to result in an exponentialdistribution of variances explained by principal components), andconvolved with an exponentially decaying GECI response kernel (meandecay time constant 1.2 s). Gaussian noise was added to the resultingtraces to emulate a GECI signal-to-noise ratio (SNR) of 25.

The randomly placed neurons and the simulated GECI activity traces werethen combined to generate a time series of volumes. To account forfluctuations of the background fluorescence due to neuropil anddetection noise, a noisy background was added throughout the syntheticvolumes (SNR 25), as well as to the final simulated sensor image. Toobtain simulated sensor data in the absence of scattering, the syntheticvolumes were convolved with a numerically simulated, ballistic LFM PSF(corresponding to a 16×0.8NA water dipping objective). To obtain anapproximation of the scattered sensor data, the synthetic volumes wereconvolved with a simulated scattered PSF obtained from a Monte-Carloapproach for a scattering length of 100 μm, a depth 400 μm, and aHenyey-Greenstein anisotropy parameter 0.9, in accordance withliterature values_(7,8).

Monte-Carlo Simulation of Scattered PSF

To generate the scattered PSFs, a Monte-Carlo approach was followedusing 100000 virtual rays launched from a point source on the opticalaxis and propagated by sampling the distances between scattering events(free paths) from an exponential distribution and scattering angles froma Henyey-Greenstein distribution. For each scattering event, a “virtual”source was placed at the apparent origin of the scattered ray and at adepth corresponding to the free path before the scattering event. Theresulting volume of virtual sources was projected forward to the sensorby convolving with the ballistic PSF. This was repeated for everylateral and axial displacement necessary to fully capture the spatiallyvarying, but periodic structure of the LFM PSF.

Statistical Analysis of Sid-Extracted Neuronal Signals

To obtain the extraction quality characterizations, a set ofsingle-plane, simultaneous 2PM-SID movies were recorded at a series ofdepths from the posterior parietal cortex of awake, head-fixed mice(100-375 μm, total n=18 recordings, 4 animals).

Signal Extraction and Tuning of Detection Characteristics

The constrained matrix factorization algorithm for Ca²⁺ signalextraction5 implemented in the CaImAn analysis package was used toanalyze the 2PM recordings, exactly implemented in the demo script₆ thatcomes with the package, thereby adapting the neuron size and approximatenumber of active neurons to values suitable for the data. After runningan initialization subroutine and the core constrained matrixfactorization, the script performs post-selection of ROIs based onspatial shape and size. It was found that the overall sensitivity andprecision of the algorithm depends mostly on the thresholds for requiredconvexity and size of neurons, as well as the approximate number ofactive neurons chosen initially. It was determined that three sets ofparameter values for the data that result in three estimation qualities:a “sensitive” estimate (avoid missing neurons while accepting a greaterrisk of detecting false positives), a “conservative” estimate (avoidfalse positives while taking greater risk of missing actual neurons),and a “balanced” setting that aims for the optimal trade-off betweensensitivity and precision.

The light-field raw data was processed. After background-subtraction,the motion metric was calculated, and motion-affected frames excludedfrom further processing. The sensitivity and precision values of SID aretuned by varying two parameters that estimate the noise floor and thebackground level, respectively, of the data and manually inspecting theoutput of the segmentation step. Sensitivity can be increased at theexpense of precision by the lowering noise floor and backgroundestimates, and vice versa. Again, three different sets of parameterswere chosen that resulted in conservative, balanced and sensitive signalextraction qualities. SID was run with the “balanced” setting on alldatasets and, in addition, with the “conservative” and “hypersensitive”settings on the recordings from one animal.

Compilation of Ground Truth and Categorization of Detections

The output of the sensitive CaImAn runs were manually inspected and thedetections contained therein were categorized as true or false positivesby assessing the shape of the detected object, and whether a singleobject was segmented into several ROIs. Any neurons that were not pickedup were added manually and categorized as false negatives. Together, thetrue positive CaImAn detections and manually added neurons (positionsand signals) in the 2PM recordings constitute what was regarded as theground truth for all further analyses.

In a second manual step, all SID runs of the “sensitive” quality settingwere assessed by comparing SID-detected locations to the ground truthlocations, identifying the matching pairs, and adding any missingneurons, marking them as false negatives. The categorizations astrue/false positives/negatives of all other CaImAn and SID results(i.e., the “balanced” and “conservative” extraction qualities) wereinferred by automatic comparison to the locations and signals that werecategorized manually based on the “sensitive” extraction output,followed by manual inspection and verification.

Neuron Detection Scores

To describe the neuron detection performance of the CaImAn and SID,three standard quality scores commonly used in the context ofclassification/detection models were computed: The score known as recallor sensitivity (ratio of true neurons to detected neurons); theprecision (ratio of true positives to total detections, i.e. to the sumof true and false positives); and the F-score, which is defined as theharmonic mean of precision and recall (multiplied by two to scale itsvalue to the (0,1) range). The F-score is one when both sensitivity andprecision are equal to one, that is, all true neurons were detectedcorrectly, and no false positives detections appeared.

These three scores for both SID and CaImAn, and the three extractionquality settings, were plotted. While the “sensitive” quality settingmaximizes the sensitivity scores in both SID and CaImAn, the“conservative” setting results in maximal precision scores. The F-scoresare optimized for the “balanced” setting. This result verifies that theparameter sets were chosen in an appropriate way, and it was determinedthat the “balanced” SID setting to be the default setting in the SIDimplementation.

Correlation Analysis of SID-Extracted Neuronal Signals

For the signal quality assessments presented in FIG. 3b , the zero-lagcorrelation coefficients of the true positive SID signals and theirrespective counterparts in the ground truth were computed, includingtheir entire duration. The values given in FIG. 3b , therefore, containinformation both about whether any peaks in the extracted signals matchwith the ground truth peaks (true/false positive GECI transientdetections), and on whether their absence in the extracted signal iscorrect (true/false negative transient detections). For comparison, alsocalculated was the correlation of the SID signals to ground truth acrosspeaks only. A histogram of the resulting peak-gated signal correlationsversus depth was made. In comparison with the ungated data shown inFIGS. 3b-i , no significant differences were observed. This is anindication that any mismatches in the extracted signals compared toground truth are not strongly biased towards false negative or falsepositive peaks, and that the ungated correlation values used throughoutFIG. 3b are a good measure of signal extraction quality.

Neuropil Rejection

Generally, it can be desirable to decontaminate the neuronal signalsfrom that of nearby neurites, as well as from any background signals(neuropil). In the disclosed embodiments, diffuse fluorescence fromneuropil and very small neurites are rejected to a large degree due tobackground subtraction and the use of a standard deviation image as thestarting point for segmentation but also the remainder of the algorithm.A planar movie from mouse cortex recorded simultaneously in LFM and 2PMwas made. While the signal-to-background ratio is as low as ˜2 in themean image of a 2PM planar movie recorded depth 200 μm, it is as high as˜20 in the standard deviation image of the same movie. In the latter,diffuse background is strongly suppressed compared to the active cellbodies and larger neurites. The high-intensity regions of the 2PMstandard deviation image, which clearly are somata, also stand out inthe corresponding reconstructed standard deviation image of the LFMrecording and reliably get identified by a local maximum searchalgorithm followed by a segmentation. This algorithm primarily picks outthe active somata, but also some of the larger and very active neurites.These larger neurites are processed further, and their spatial andtemporal components are optimized iteratively as described above. Afterthe optimization, the optimized spatial components can be reconstructedto more closely examine their shape. While the cell bodies are compact,larger and spherically shaped, neurites often extend over a largerregion, both due to their morphology and since nearby neurites are oftenmerged into the same spatial component due to their correlated activity,and have less regular shapes. These differences are used for manual orautomatized post-selection processing whereby the signals from neuritescan be identified and subtracted out from that of neuronal cell bodies.

Motion Detection and Correction

During imaging sessions, mice were head-fixed using a customized mountcomplemented with a head bar holder and a mouse body stabilizer (bodyjacket) and could run freely on a disc (200 mm diameter), as describedin more detail elsewhere₁₂. This considerably reduced animal-inducedmotion of the brain during imaging. To detect any residual motion in theraw SID/LFM raw data prior to further processing, a simple motiondetection metric based on image autocorrelation was developed, which iscomputed as follows. First, the raw data is background-subtracted byrank-1 non-negative matrix factorization of the time series of SID/LFMcamera frames. Next, the difference frames between allbackground-subtracted frames are computed, and the autocorrelationimages of the difference frames are computed. In the difference frames,translation of a source within the FOV manifests itself as negativevalues at pixels illuminated from the previous source position, andpositive values at pixels illuminated by the new source position. Hence,the values of these two sets of pixels will be anti-correlated,resulting in a negative peak in the autocorrelation image, at a spatial“lag” (distance) corresponding to the extent of the motion effect. Theminima of each autocorrelation image (normalized to the maximum of theautocorrelation image) were extracted, and the time derivative of thisseries of minima was taken to obtain a clear metric for motion in theLFM raw frames. This metric was plotted for data from a simultaneous2PM+SID recording. The motion metric computed from the of the 2PM andLFM/SID raw data are in good agreement, and the peaks in both metricscorrelate with the onset of animal motion as recorded by tracking themovement of the running disc with a high-resolution optical computermouse.

In SID/LFM, the point-spread function of the system is engineered tovary spatially (in order to provide axial resolution), so a translationof a source does not result in a mere translation of the image on thesensor as in classical wide-field imaging, but a more intricatetransformation. However, it was found that simply taking the minima ofthe difference frame autocorrelation images still picks up motion well.

Pixels affected by motion would exhibit high standard deviation alongtime that does not originate from neuronal activity, and would thusnegatively affect the precision of SID demixing and segmentation.Therefore, frames with a motion metric value above a threshold wereexcluded prior to computing the standard deviation image (step ii inFIG. 1).

Neural activity from the motion-affected frames was not recovered. SinceLFM/SID captures the full recording volumes in an unbiased way, it wasexpected to be possible to recover neuron activity information byregistering the SID-detected neuron footprints of the unaffected framesto the transformed footprints in the motion-affected frames and extractthe source brightness. As mentioned above, the translation of a source(neuron) in LFM/SID results in transformation of its LFM image that isnot a simple translation, due to the spatially varying point-spreadfunction in LFM. However, since the point-spread function is known, itis possible to map source positions to images and iteratively find thetransformation of source positions that best explains the image observedduring motion frames. This procedure can be based on a standardoptimizer for image registration, with the additional step of mappingposition estimates to LFM images by convolving with the LFM point-spreadfunction.

Optical Alignment of MiniLFM

For the conversion of a conventional widefield Miniscope to a MiniLFM, amicrolens array was introduced in the optical path at the image plane,and exactly one focal length away from the CMOS imaging sensor. In oneexample, the microlens array has a focal length of 780 μm and measured13×13 mm with a lenslet pitch of 100 μm (RPC Photonics MLA-S-100-f8). Tobe able to position it at a distance of 780 μm from the active surfaceof the image sensor, the sensor cover glass was removed by charring theglue that holds it in place using a hot air soldering rework station.

To accurately position the CMOS imaging sensor (1280×1024 pixels, 5.2 μmpixel size; ON Semiconductor, USA) in the back focal plane of themicrolens array, custom-made holders were employed for both elements. Incombination with a three-axis translation stage and high-precisionkinematic mounts (Thorlabs Inc., USA), the setup allowed fortranslation, rotation and tilt in six degrees of freedom at micrometerprecision. An expanded, collimated green laser beam (532 nm) wasdirected at normal incidence onto the MLA, and the relative position ofMLA and sensor adjusted until the sensor image showed optimal anduniform focal spots behind each microlens.

In an iterative process, the focal spots were analyzed using an ImageJmacro (Supplementary Software), and alignment was adjusted accordingly.MLA rotation was diagnosed simply by plotting line profiles across theframe; tilt and translation were quantified via particle analysis. Thearea of the individual focused laser spots in pixels, and the meanintensity per spot, were plotted in real time to visualize focalposition and tilt in a color-coded way for all 3600 spots across the FOV(Supplementary Software). A homogeneous distribution of peak focal spotintensity across the frame indicates absence of tilt. Further, the areaof the laser spots is smallest when the sensor is placed in the focalplane of the microlens array. Additionally, individual spots of thewell-aligned system across the FOV were examined for size, intensity andsymmetry.

The results from particle analysis were thus used to determine theprecise position of the elements at which a simultaneous minimum offocal spot area and a maximum of mean intensity was reached. Once thisconfiguration was obtained, the components were permanently glued toeach other with high viscosity UV-curing adhesive (NOA63, Norland, USA)under a stereomicroscope.

To achieve a well-defined magnification and object-space workingdistance in spite of variations in the spacing of GRIN objective andtube lens, the microscope was adjusted to operate in “infinity”configuration. In a non-LFM microscope, this means that the image sensoris placed in the back focal plane of the tube lens. In an LFM, thistranslates to the MLA being placed in the back focal plane of the tubelens (and the sensor in the back focal plane of the MLA, as guaranteedby the alignment procedure described above). To find the “infinity”configuration, a collimated green laser is aimed through an iris andinto the bottom opening of the MiniLFM, without the GRIN objective inplace. The laser passes through the filters, gets focused by the tubelens, and a fraction of its intensity is reflected from the surface ofthe MLA and propagates back through the previous elements. Now, thedistance of the MLA from the tube lens is adjusted until theback-reflection of the laser from the surface of the MLA emerging fromthe bottom opening of the MiniLFM is collimated. This is the case onlyif the reflecting surface (the MLA) is located in the back focal planeof the tube lens.

Miniature Head-Mounted Light-Field Microscope.

The MiniLFM design is based on the open source Miniscope project_(23A):Blue light from an LED is collimated by a ball lens, passed through anexcitation filter (Chroma ET470/40×), and reflected off a dichroicmirror (Chroma T495lpxr). A GRIN lens (Edmund 64-520, 0.5NA, 0.23 pitch,diameter 1.8 mm, length 3.93 mm, working distance at 530 nm: approx. 200μm) is implanted surgically such that its focal plane coincides with theaxial center of the sample region of interest (see below for surgicalprocedures). Excitation light passes through the GRIN lens, which alsocollects fluorescence light. Fluorescence then passes through thedichroic mirror, an emission filter (Chroma ET525/50m), and anachromatic doublet tube lens (Edmund 45-207, f=15 mm) that forms an8.93-fold magnified image of the GRIN front focal plane. An MLA (RPCPhotonics MLA-S-100-f8, f=780 μm, microlens pitch 100 μm, squarepattern, no gaps, diced to 13×13 mm, 2 mm substrate thickness) is placedin this image plane, and the image sensor (On SemiconductorMT9M001C12STM, 1.3 Mpx, 5.2 μm pixel size, rolling shutter) in the focalplane of the MLA. To accommodate the microlens array, the part holdingthe image sensor was elongated by 2.7 mm compared to the Miniscopedesign. The MLA and sensor are aligned w.r.t. each other using a customalignment rig and glued together using UV-curing glue. To guarantee aknown magnification, the distance of the GRIN and tube lenses is fixedsuch that the two lenses are placed at the sum of their focal lengths.Readout electronics, firmware and software do not differ from thosepublished by the Miniscope project. The full frame readout time of thesensor chip is 50 ms, which is short compared to the GCaMP6f rise time(200 ms); the effects of the rolling shutter readout pattern on neurontiming extraction therefore are negligible. It is noted that overallminiscope weight can be reduced in the future by using a custom MLA witha thinner glass substrate (0.2 mm available from same manufacturer).This would reduce overall weight by ˜15%. To improve stability of theMiniLFM relative to the baseplate, one facet of the MiniLFM body basewas reinforced with a thin 1×1.5 mm aluminum plate to allow for morerigid fixation to the baseplate with a setscrew. Stability can beimproved further by using removable adhesives (such as siliconeelastomers, the weight of which is negligible) to connect the body tothe baseplate.

Signal Extraction and Data Analysis.

Raw data was processed using a pipeline based on the recentlyestablished SID algorithm 4, which is briefly outlined in the following:After rank-1 matrix factorization for background subtraction, a motionmetric based on the value range of the difference frames is calculated.The time series of raw frames is split at all time points where themotion metric exceeds a threshold, and the resulting low-motion segmentsare processed separately using the SID algorithm. For each of thesegments, the standard deviation image is calculated, reconstructed byconstrained deconvolution with a simulated PSF of the system, andsegmented using a local maximum search. The resulting neuron candidatelocations are used to seed a dictionary of spatial footprint templatesthat are iteratively updated using a constrained spatio-temporal matrixfactorization algorithm that alternatingly updates the temporal(spatial) components, while keeping the spatial (temporal) componentsfixed. This results in a set of neuron footprints (i.e., the set ofimages of each neuron on the LFM sensor) and temporal signals. Theneuron footprints are reconstructed individually by deconvolution withthe aforementioned simulated LFM PSF of the optical system. Thesereconstructed, volumetric images of each neuron are checked for spatialcompactness and compatibility with an expected neuron size.Subsequently, the neuron footprints and temporal signals from all thelow-motion segments are pooled (merging neurons with stronglyoverlapping footprints). The temporal signals at this stage may stillexhibit short glitches due to weaker motion events. These glitchesexhibit sudden rises or drops in neuron brightness, lasting approx. 1-10frames, and synchronized across most signals. These motion glitches weredetected using the motion metric mentioned above (with optional manualadditions) and interpolate the signals across the glitches by learning amodel of GECI response dynamics_(31A) on each neuron and using it tointerpolate across the motion-affected frames. The same GECI responsemodel also yields the estimate of underlying firing rate. Since themodel does not take into account a calibration of relative fluorescencechange to underlying action potentials, the resulting calciumconcentration and firing rate estimates are quoted in arbitrary units.

Simultaneous Two-Photon Microscopy and MiniLFM Recordings

In order to verify MiniLFM/SID results by comparison with simultaneouslyacquired two-photon microscopy data, awake mice (expressing GCaMP6f inhippocampus CA1, with implanted GRIN lens, and with a metal headbar andMiniLFM baseplate attached to the skull; see below for animalprocedures) were mounted head-fixed but free to walk on a circulartreadmill assembly_(11A) that allowed for precise positioning andalignment of the mouse head. A modified MiniLFM device was interfacedwith a commercial upright two-photon microscope (2PM; ScientificaSlicescope with Coherent Chameleon Ultra II laser tuned to 920 nm,Olympus PlanApo N 1.25×/0.04 objective). The MiniLFM body was cut at thelocation of the fluorescence emission path, and a beam splitter(Thorlabs BST10R), which transmits 2P excitation light and reflects 70%of the GCaMP emission, was incorporated at that location, mounted at a45-degree angle w.r.t. to the optical axis. The reflected GCaMP emissionwas passed through two infrared blocking filters (Thorlabs GFS900-A andSemrock Brightline 720SP) to remove 2P excitation light, and directedonto an unmodified MiniLFM detection module, consisting of a microlensarray aligned and glued to a CMOS sensor, as described above.Transmitted GCaMP emission was directed into the 2PM objective anddetected on a photomultiplier-tube in the Slicescope non-descanneddetection arm. MiniLFM frame rate was set to 2 Hz, and the 2PMacquisition trigger synchronized to the MiniLFM frame clock. The 2PM wasset to acquire and average 9 frames for each MiniLFM frame to maximizefluorescence excitation.

A total of n=5 recordings was acquired from two mice, lasting 180 seach. The MiniLFM data was processed using the SID algorithm, asdescribed above. The 2PM data was passed through the CaImAnalgorithm_(31A) to detect active neurons and extract their signals.CaImAn output was inspected manually and corrected for false positiveand false negative detections to establish a human-verified groundtruth. The SID detected neurons were then compared to the ground truthand classified as true/false positives/negatives, and correlationsbetween paired SID & ground-truth temporal signals were calculated. Inaddition, excess mutual information was calculated as the differencebetween the mutual information figure for each possible pair of groundtruth neuronal activity traces, and the corresponding pairs of SIDactivity traces.

Quantification of Animal Agility

Mice were trained (for five consecutive days) to run back and forth onan elevated linear track (37 cm above ground, 198 cm long, wall height 2cm) for water rewards offered in “base” areas at either end of thetrack. After training was completed, mouse behavior was recorded usingan overhead camera (HD webcam C615, Logitech) for each of the threeconditions (no device mounted, with Miniscope, with MiniLFM). One triallasted 10 minutes, three trials were carried out per day for each of thethree mice (one trial for each condition, in permuted order) withinter-trial resting periods of one hour. Trials were repeated for threeconsecutive days, resulting in a total of n=27 trials. Videos wereanalyzed by manually evaluating the number of times the animals wouldtraverse the track and counting the number of stops. Speed wascalculated by measuring the distance travelled along the track using ascreen ruler, and dividing this value by the time required for thetransversal (not including stops).

Quantification of Acceleration Due to Motion and Motion Artefacts

To measure the acceleration experienced by the MiniLFM head-mounteddevice, a circuit board containing a three-axis MEMS accelerometer chip(Sparkfun ADXL335, range ±3 g, 10 bits per axis, 50 Hz bandwidth) wasattached to the back of the MiniLFM sensor circuit board. It wasconnected via five thin wires to an Arduino microcontroller, which readout the raw acceleration values and transferred them to a PC. The rawvalues were high-pass filtered to remove the effects of gravity andbinned to match the MiniLFM frame rate.

Motion artefacts in widefield Miniscope recordings were quantified byapplying the recursive, FFT-based rigid image registration algorithmpublished as part of the Miniscope data analysis package athttps://github.com/daharoni/Miniscope_Analysis.

Experimental Model and Subject Details

All procedures were in accordance with the Institutional Animal Care andUse Committee (IACUC) at The Rockefeller University, New York. Mice wereobtained from The Jackson Laboratory (C57BL/6J) and typicallygroup-housed with a 12 h/12 h light cycle in standard cages, with foodand water ab libitum.

Animal Surgery and In-Vivo Ca²⁺ Imaging of Freely Moving Mice.

Adult (P90+) male and female C57Bl/6J wild-type mice (n=5) wereanesthetized with isoflurane (1-1.5%, flow rate 0.5-0.7 l/min) andplaced in a stereotactic frame (RWD Life Science Co., Ltd., China). 250nl of AAV1.Syn.GCaMP6f.WPRE.SV40 (titer ˜10¹² viral particles/ml,AV-1-PV2822 Penn Vector Core) was injected in the posterior hippocampus,coordinates 2.1 mm posterior to bregma, 2 mm lateral and −1.65 mmdorsoventral from the top of the skull. Nucleus-localizedAAV9.Syn.H2B.GCaMP6f.WPRE.Pzac2.1 was injected at the same titer.Injections were made with a microinjection controller (World PrecisionInstruments, FL) using glass pipettes previously pulled and beveled,filled with mineral oil. One week after injection, the GRIN lensimplantation surgery was made. After removing the scalp and clearing theskull of connective tissues, a custom-made lightweight metal headbar wasfixed onto the skull with cyanoacrylate adhesive (Krazy Glue) andcovered with black dental cement (Ortho-Jet, Lang Dental, USA). Theoutline of the craniotomy was made using the injection site as areference. From the injection site, the midpoint of the craniotomy wasset 0.5 mm closer to bregma. After removing the skull, the cortex wasaspirated with abundant cold saline solution until the corpus callosumbecame visible, and the horizontal striations were carefully removeduntil vertical striations became visible. When the entire area was cleanand the bleeding had stopped, the GRIN lens was slowly inserted, to adepth of 1.35 mm from the top of the skull and glued in place usingVetbond (3M). When dry, the rest of the skull was covered with blackdental cement. To prevent post-surgical infections and post-surgicalpain, mice were fed pellets with antibiotic supplement (trimethoprim andsulfamethoxazole, Purina Mod 5053, LabDiet, MO) for 2 weeks and 1 mg/mlmeloxicam i.p. injections (Putney, UK) for 3 to 5 days. Two weeks afterthe last surgery, the mice were anesthetized and placed in thestereotactic frame again, for affixing the baseplate of the miniaturemicroscope. To this end, the baseplate is attached to the MiniLFM andthe alignment of the baseplate orientation is adjusted manually untilthe illuminated FOV is centered on the image sensor, and the brightcircles formed from diffuse illumination by the microlens array on thesensor appear symmetrical w.r.t. the center of the FOV. The baseplate isthen glued in place using dental cement and Krazy Glue. The MiniLFM isremoved as soon as the dental cement has hardened, and the animalreturned to its home cage. After this, the animal is ready for imaging.

Imaging was done in experimental sessions lasting no longer than onehour. The MiniLFM was snapped onto the affixed baseplate, where it getsheld in place by small magnets embedded in the baseplate as well as thebottom face of the MiniLFM, and additionally locked by a setscrew. Themice were placed into an open field arena or into a linear track wherethey walked freely during the recording session.

A total of 12 neuronal recordings from 5 animals were analyzed(including simultaneous 2PM-MiniLFM verification recordings). Animalswere included in the study for which all preparatory animal proceduresworked sufficiently well to allow for signal detection. Provided thatanimal procedures (surgeries and viral injections/GECI expression) weresuccessful as verified using a standard two-photon microscope, imagingresults and data quality were found to be reliably reproducible, bothacross imaging sessions with the same animal, and across animals. Sincethe object of this study is to establish a neural recording methodrather than any biological findings, this sample size is sufficient toverify the performance of the disclosed method.

Only animals were included in the study for which all preparatory animalprocedures worked sufficiently well to allow for signal detection (i.e.,GECI expression observable, implanted GRIN lens placement correct), asverified using a standard two-photon microscope. Of these animals, nonewere excluded.

For all animals in which animal procedures (surgeries and viralinjections/GECI expression) were successful (as verified using astandard two-photon microscope), imaging and data analysis results werereliably reproduced, both across imaging sessions with the same animal,and across animals.

Software and Computing Systems

Custom code for the MiniLFM alignment and Data analysis pipeline wasdeveloped. Custom-written Java (ImageJ/Fiji, release 2017 May 30) and R(v3.x) code implementing focal spot analysis for LFM alignment, as wellas Matlab (2017a) code implementing the signal extraction and motiondetection pipeline, as described in the Main Text and Online Methodswere also developed. The SID Matlab package published as SupplementarySoftware with a prior publication Nöbauer, T. et al. Video ratevolumetric Ca²⁺ imaging across cortex using seeded iterative demixing(SID) microscopy. Nat Meth 14, 811-818 (2017), doi:10.1038/nmeth.4341,is required, as well as the dependencies listed in the README.txt fileaccompanying that package.

One or more embodiments disclosed herein, or a portion thereof, may makeuse of software running on a computer or workstation. By way of example,only and without limitation, FIG. 4 is a block diagram of an embodimentof a machine in the form of a computing system 400, within which is aset of instructions 402 that, when executed, cause the machine toperform any one or more of the methodologies according to embodiments ofthe disclosed subject matter. In one or more embodiments, the machineoperates as a standalone device; in one or more other embodiments, themachine is connected (e.g., via a network 422) to other machines. In anetworked implementation, the machine operates in the capacity of aserver or a client user machine in a server-client user networkenvironment. Exemplary implementations of the machine as contemplated byembodiments of the disclosed subject matter include, but are not limitedto, a server computer, client user computer, personal computer (PC),tablet PC, personal digital assistant (PDA), cellular telephone, mobiledevice, palmtop computer, laptop computer, desktop computer,communication device, personal trusted device, web appliance, networkrouter, switch or bridge, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine.

The computing system 400 includes a processing device(s) 404 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU), orboth), program memory device(s) 406, and data memory device(s) 408,which communicate with each other via a bus 410. The computing system400 further includes display device(s) 412 (such as a liquid crystaldisplay (LCD), flat panel, solid state display, or cathode ray tube(CRT)). The computing system 400 includes input device(s) 414 (e.g., akeyboard), cursor control device(s) 416 (e.g., a mouse), disk driveunit(s) 418, signal generation device(s) 420 (e.g., a speaker or remotecontrol), and network interface device(s) 424, operatively coupledtogether, and/or with other functional blocks, via bus 410.

The disk drive unit(s) 418 includes machine-readable medium(s) 426, onwhich is stored one or more sets of instructions 402 (e.g., software)embodying any one or more of the methodologies or functions herein,including those methods illustrated herein. The instructions 402 mayalso reside, completely or at least partially, within the program memorydevice(s) 406, the data memory device(s) 408, and/or the processingdevice(s) 404 during execution thereof by the computing system 400. Theprogram memory device(s) 406 and the processing device(s) 404 alsoconstitute machine-readable media. Dedicated hardware implementations,such as but not limited to ASICs, programmable logic arrays, and otherhardware devices can likewise be constructed to implement methodsdescribed herein. Applications that include the apparatus and systems ofvarious embodiments broadly comprise a variety of electronic andcomputer systems. Some embodiments implement functions in two or morespecific interconnected hardware modules or devices with related controland data signals communicated between and through the modules, or asportions of an ASIC. Thus, the example system is applicable to software,firmware, and/or hardware implementations.

The term “processing device” as used herein is intended to include anyprocessor, such as, for example, one that includes a CPU (centralprocessing unit) and/or other forms of processing circuitry. Further,the term “processing device” may refer to more than one individualprocessor. The term “memory” is intended to include memory associatedwith a processor or CPU, such as, for example, RAM (random accessmemory), ROM (read only memory), a fixed memory device (for example,hard drive), a removable memory device (for example, diskette), a flashmemory and the like. In addition, the display device(s) 412, inputdevice(s) 414, cursor control device(s) 416, signal generation device(s)420, etc., can be collectively referred to as an “input/outputinterface,” and is intended to include one or more mechanisms forinputting data to the processing device(s) 404, and one or moremechanisms for providing results associated with the processingdevice(s). Input/output or I/O devices (including but not limited tokeyboards (e.g., alpha-numeric input device(s) 414, display device(s)412, and the like) can be coupled to the system either directly (such asvia bus 410) or through intervening input/output controllers (omittedfor clarity).

In an integrated circuit implementation of one or more embodiments ofthe disclosed subject matter, multiple identical die are typicallyfabricated in a repeated pattern on a surface of a semiconductor wafer.Each such die may include a device described herein, and may includeother structures and/or circuits. The individual dies are cut or dicedfrom the wafer, then packaged as integrated circuits. One skilled in theart would know how to dice wafers and package die to produce integratedcircuits. Any of the exemplary circuits or method illustrated in theaccompanying figures, or portions thereof, may be part of an integratedcircuit. Integrated circuits so manufactured are considered part of thisdisclosed subject matter.

An integrated circuit in accordance with the embodiments of thedisclosed subject matter can be employed in essentially any applicationand/or electronic system in which buffers are utilized. Suitable systemsfor implementing one or more embodiments of the disclosed subject matterinclude, but are not limited, to personal computers, interface devices(e.g., interface networks, high-speed memory interfaces (e.g., DDR3,DDR4), etc.), data storage systems (e.g., RAID system), data servers,etc. Systems incorporating such integrated circuits are considered partof embodiments of the disclosed subject matter. Given the teachingsprovided herein, one of ordinary skill in the art will be able tocontemplate other implementations and applications.

In accordance with various embodiments, the methods, functions or logicdescribed herein is implemented as one or more software programs runningon a computer processor. Dedicated hardware implementations including,but not limited to, application specific integrated circuits,programmable logic arrays and other hardware devices can likewise beconstructed to implement the methods described herein. Further,alternative software implementations including, but not limited to,distributed processing or component/object distributed processing,parallel processing, or virtual machine processing can also beconstructed to implement the methods, functions or logic describedherein.

The embodiment contemplates a machine-readable medium orcomputer-readable medium containing instructions 402, or that whichreceives and executes instructions 402 from a propagated signal so thata device connected to a network environment 422 can send or receivevoice, video or data, and to communicate over the network 422 using theinstructions 402. The instructions 402 are further transmitted orreceived over the network 422 via the network interface device(s) 424.The machine-readable medium also contains a data structure for storingdata useful in providing a functional relationship between the data anda machine or computer in an illustrative embodiment of the systems andmethods herein.

While the machine-readable medium 402 is shown in an example embodimentto be a single medium, the term “machine-readable medium” should betaken to include a single medium or multiple media (e.g., a centralizedor distributed database, and/or associated caches and servers) thatstore the one or more sets of instructions. The term “machine-readablemedium” shall also be taken to include any medium that is capable ofstoring, encoding, or carrying a set of instructions for execution bythe machine and that cause the machine to perform anyone or more of themethodologies of the embodiment. The term “machine-readable medium”shall accordingly be taken to include, but not be limited to:solid-state memory (e.g., solid-state drive (SSD), flash memory, etc.);read-only memory (ROM), or other non-volatile memory; random accessmemory (RAM), or other re-writable (volatile) memory; magneto-optical oroptical medium, such as a disk or tape; and/or a digital file attachmentto e-mail or other self-contained information archive or set of archivesis considered a distribution medium equivalent to a tangible storagemedium. Accordingly, the embodiment is considered to include anyone ormore of a tangible machine-readable medium or a tangible distributionmedium, as listed herein and including art-recognized equivalents andsuccessor media, in which the software implementations herein arestored.

It should also be noted that software, which implements the methods,functions and/or logic herein, are optionally stored on a tangiblestorage medium, such as: a magnetic medium, such as a disk or tape; amagneto-optical or optical medium, such as a disk; or a solid statemedium, such as a memory automobile or other package that houses one ormore read-only (non-volatile) memories, random access memories, or otherre-writable (volatile) memories. A digital file attachment to e-mail orother self-contained information archive or set of archives isconsidered a distribution medium equivalent to a tangible storagemedium. Accordingly, the disclosure is considered to include a tangiblestorage medium or distribution medium as listed herein and otherequivalents and successor media, in which the software implementationsherein are stored.

Although the specification describes components and functionsimplemented in the embodiments with reference to particular standardsand protocols, the embodiment are not limited to such standards andprotocols.

The illustrations of embodiments described herein are intended toprovide a general understanding of the structure of various embodiments,and they are not intended to serve as a complete description of all theelements and features of apparatus and systems that might make use ofthe structures described herein. Many other embodiments will be apparentto those of skill in the art upon reviewing the above description. Otherembodiments are utilized and derived therefrom, such that structural andlogical substitutions and changes are made without departing from thescope of this disclosure. Figures are also merely representational andare not drawn to scale. Certain proportions thereof are exaggerated,while others are decreased. Accordingly, the specification and drawingsare to be regarded in an illustrative rather than a restrictive sense.

Such embodiments are referred to herein, individually and/orcollectively, by the term “embodiment” merely for convenience andwithout intending to voluntarily limit the scope of this application toany single embodiment or inventive concept if more than one is in factshown. Thus, although specific embodiments have been illustrated anddescribed herein, it should be appreciated that any arrangementcalculated to achieve the same purpose are substituted for the specificembodiments shown. This disclosure is intended to cover any and alladaptations or variations of various embodiments. Combinations of theabove embodiments, and other embodiments not specifically describedherein, will be apparent to those of skill in the art upon reviewing theabove description.

In the foregoing description of the embodiments, various features aregrouped together in a single embodiment for the purpose of streamliningthe disclosure. This method of disclosure is not to be interpreted asreflecting that the claimed embodiments have more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive subject matter lies in less than all features of asingle embodiment. Thus the following claims are hereby incorporatedinto the detailed description, with each claim standing on its own as aseparate example embodiment.

The abstract is provided to comply with 37 C.F.R. § 1.72(b), whichrequires an abstract that will allow the reader to quickly ascertain thenature of the technical disclosure. It is submitted with theunderstanding that it will not be used to interpret or limit the scopeor meaning of the claims. In addition, in the foregoing DetailedDescription, it can be seen that various features are grouped togetherin a single embodiment for the purpose of streamlining the disclosure.This method of disclosure is not to be interpreted as reflecting anintention that the claimed embodiments require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive subject matter lies in less than all features of asingle embodiment. Thus the following claims are hereby incorporatedinto the Detailed Description, with each claim standing on its own asseparately claimed subject matter.

Although specific example embodiments have been described, it will beevident that various modifications and changes are made to theseembodiments without departing from the broader scope of the inventivesubject matter described herein. Accordingly, the specification anddrawings are to be regarded in an illustrative rather than a restrictivesense. The accompanying drawings that form a part hereof, show by way ofillustration, and without limitation, specific embodiments in which thesubject matter are practiced. The embodiments illustrated are describedin sufficient detail to enable those skilled in the art to practice theteachings herein. Other embodiments are utilized and derived therefrom,such that structural and logical substitutions and changes are madewithout departing from the scope of this disclosure. This DetailedDescription, therefore, is not to be taken in a limiting sense, and thescope of various embodiments is defined only by the appended claims,along with the full range of equivalents to which such claims areentitled.

Given the teachings provided herein, one of ordinary skill in the artwill be able to contemplate other implementations and applications ofthe techniques of the disclosed embodiments. Although illustrativeembodiments have been described herein with reference to theaccompanying drawings, it is to be understood that these embodiments arenot limited to the disclosed embodiments, and that various other changesand modifications are made therein by one skilled in the art withoutdeparting from the scope of the appended claims.

REFERENCES

-   1. Chen et al., Nature 499, 295-300 (2013).-   2. Schrödel et al., Nat. Methods 10, 1013-1020 (2013).-   3. Nguyen et al., Proc. Natl. Acad. Sci. 113, E1074-E1081 (2016).-   4. Prevedel et al., Nat. Methods 11, 727-730 (2014).-   5. Ahrens et al., Nat. Methods 10, 413-420 (2013).-   6. Denk et al., Science 248, 73-76 (1990).-   7. Helmchen et al., Nat. Methods 2, 932-940 (2005).-   8. Ji et al., Nat. Neurosci. 19, 1154-1164 (2016).-   9. Grewe et al., Nat. Methods 7, 399-405 (2010).-   10. Botcherby et al., Proc. Natl. Acad. Sci. 109, 2919-2924 (2012).-   11. Kirkby et al., Opt. Express 18, 13720 (2010).-   12. Cheng et al., Nat. Methods 8, 139-142 (2011).-   13. Kim et al., Opt. Express 15, 11658 (2007).-   14. Stirman et al., Wide field-of-view, twin region two-photon    imaging across extended cortical networks. (2014). at    http://biorxiv.org/lookup/doi/10.1101/011320-   15. Reddy et al., Nat. Neurosci. 11, 713-720 (2008).-   16. Katona et al., Nat. Methods 9, 201-208 (2012).-   17. Fernández-Alfonso et al., J. Neurosci. Methods 222, 69-81    (2014).-   18. Prevedel et al., Nat. Methods advance online publication,    (2016).-   19. Huisken, J. Science 305, 1007-1009 (2004).-   20. Wu, et al., Proc. Natl. Acad. Sci. 108, 17708-17713 (2011).-   21. Chhetri et al., Nat. Methods 12, 1171-1178 (2015).-   22. Bouchard et al., Nat. Photonics 9, 113-119 (2015).-   23. Yang, et al, Neuron 89, 269-284 (2016).-   24. Abrahamsson et al., Nat. Methods 10, 60-63 (2012).-   25. Packer et al., Nat. Methods 12, 140-146 (2015).-   26. Levoy et al., ACM Trans. Graph. 25, 924 (2006).-   27. Broxton et al., Opt. Express 21, 25418-25439 (2013).-   28. Cohen et al., Opt. Express 22, 24817-24839 (2014).-   29. Pégard et al., Optica 3, 517 (2016).-   30. Liu et al., Opt. Express 23, 14461 (2015).-   31. Kaifosh et al., Front. Neuroinformatics 8, 80 (2014).-   32. Pachitariu et al., Extracting regions of interest from    biological images with convolutional sparse block coding. in Adv.    Neural Inf. Process. Syst. 1745-1753 (2013). at    <http://papers.nips.cc/paper/5167-extracting-regions-of-interest-from-biological-imageswith-convolutional-sparse>-   33. Mukamel et al., Neuron 63, 747-760 (2009).-   34. Pnevmatikakis et al., Neuron (2016). at    http://www.sciencedirect.com/science/article/pii/S0896627315010843-   35. Maruyama et al., Neural Netw. Off. J. Int. Neural Netw. Soc. 55,    11-19 (2014).-   36. Haeffele et al., Structured Low-Rank Matrix Factorization:    Optimality, Algorithm, and Applications to Image Processing. in    Proc. 31st Int. Conf. Mach. Learn. 32, (JMLR: 2014).-   37. Diego Andilla, F. & Hamprecht, F. A. in Adv. Neural Inf.    Process. Syst. 27 (eds. Ghahramani, Z., Welling, M., Cortes, C.,    Lawrence, N. D. & Weinberger, K. Q.) 64-72 (Curran Associates, Inc.,    2014). at    <http://papers.nips.cc/paper/5342-sparse-space-timedeconvolution-for-calcium-image-analysis.pdf>-   38. Nairat et al., Approach for incorporating aerosol scattering in    wave optics propagation simulation in 2013 IEEE Aerosp. Conf.    1-5 (2013) doi:10.1109/AERO.2013.6497321-   39. Jacques, S. L., Phys. Med. Biol. 58, R37 (2013).-   40. Dombeck et al., Nat. Neurosci. 13, 1433-1440 (2010).-   41. Kaifosh et al., Nat. Neurosci. 16, 1182-1184 (2013).-   42. Graves et al., Neuron 76, 776-789 (2012).-   43. Waller et al., Nature 523, 416-417 (2015).-   44. Zhou et al., Efficient and accurate extraction of in vivo    calcium signals from microendoscopic video data. in (2015). at    https://www.semanticscholar.org/paper/Efficient-and-accurate-extraction-of-in-vivo-Zhou-Resendez/e57db51b5518da58ae734ff58014f3a862b30b0e-   45. Apthorpe et al., Automatic Neuron Detection in Calcium Imaging    Data Using Convolutional Networks. ArXiv160607372 Cs Q-Bio (2016).    at http://arxiv.org/abs/1606.07372-   46. Wang et al., Nat. Commun. 6, (2015).-   1A. Ji et al., Nat. Neurosci. 19, 1154-1164 (2016).-   2A. Yang et al., Nat. Methods 14, 349-359 (2017).-   3A. Prevedel et al., Nat. Methods 11, 727-730 (2014).-   4A. Nöbauer et al., Nat. Methods 14, 811-818 (2017).-   5A. Pégard et al., Optica 3, 517 (2016).-   6A. Huisken et al., Science 305, 1007-1009 (2004).-   7A. Ahrens et al., Nat. Methods 10, 413-420 (2013).-   8A. Chhetri et al., Nat. Methods 12, 1171-1178 (2015).-   9A. Bouchard et al., Nat. Photonics 9, 113-119 (2015).-   10A. Schrödel et al., Nat. Methods 10, 1013-1020 (2013).-   11A. Prevedel et al., Nat. Methods 13, 1021-1028 (2016).-   12A. Duemani Reddy et al., Nat. Neurosci. 11, 713-720 (2008).-   13A. Yang et al., Neuron 89, 269-284 (2016).-   14A. Katona et al., Nat. Methods 9, 201-208 (2012).-   15A. Fernández-Alfonso, et al., J. Neurosci. Methods 222, 69-81    (2014).-   16A. Botcherby et al., Proc. Natl. Acad. Sci. 109, 2919-2924 (2012).-   17A. Lu, et al., Nat. Neurosci. 20, 620-628 (2017).-   18A. Song, et al., Nat. Methods 14, 420-426 (2017).-   19A. Chen et al., Nature 499, 295-300 (2013).-   20A. Helmchen et al., Neuron 31, 903-912 (2001).-   21A. Flusberg et al., Nat. Methods 5, 935-938 (2008).-   22A. Ghosh, et al., Nat. Methods 8, 871-878 (2011).-   23A. Cai et al., Nature 534, 115-118 (2016).-   24A. Barbera, et al., Neuron 92, 202-213 (2016).-   25A. Sabharwal et al., Appl. Opt. 38, 7133-7144 (1999).-   26A. Jung et al., J. Neurophysiol. 92, 3121-3133 (2004).-   27A. Flusberg et al., Opt. Lett. 30, 2272-2274 (2005).-   28A. Ziv et al., Nat. Neurosci. 16, 264-266 (2013).-   29A. Levoy, et al., ACM Trans. Graph. 25,924 (2006).-   30A. Broxton et al., Opt. Express 21, 25418-25439 (2013).-   31A. Pnevmatikakis et al., Neuron 89, 285-299 (2016).-   32A. Matz et al., Opt. Express 24, 10987-11001 (2016).

What is claimed is:
 1. An imaging signal extraction apparatuscomprising: an imaging apparatus interface; a processing device, theprocessing device operatively coupled to the imaging apparatusinterface; and a computer readable medium comprising instructions that,when executed by the processing device, perform operations comprising:a) generating a two-dimensional image from imaging information obtainedfrom the imaging apparatus interface, thereby estimating ballisticcomponent of the imaging information; b) generating a three-dimensionalimage by remapping the two-dimensional image; c) identifying a candidateobject in the three-dimensional image; d) obtaining an estimated spatialforward model of the candidate object by mapping the three-dimensionalimage of the candidate object with a point-spread-function associatedwith the imaging apparatus; e) obtaining background-corrected data byusing the estimated spatial forward model of the candidate object andestimated temporal components; and f) iteratively updating the estimatedspatial forward model and estimated temporal components untilconvergence is reached for the candidate object, thereby extracting thesignal information.
 2. The imaging signal extraction apparatus definedby claim 1 wherein the imaging signal extraction apparatus is alight-field microscope.
 3. The imaging signal extraction apparatusdefined by claim 1 wherein the two-dimensional image is atwo-dimensional standard deviation image.
 4. The imaging signalextraction apparatus defined by claim 1 wherein generating athree-dimensional image by remapping comprises deconvolving thetwo-dimensional image.
 5. The imaging signal extraction apparatusdefined by claim 1 wherein obtaining an estimated spatial forward modelof the candidate object by mapping comprises convolving thethree-dimensional image of the candidate object with apoint-spread-function associated with the imaging apparatus.
 6. Theimaging signal extraction apparatus defined by claim 1, wherein beforeoperation (a), background information obtained by the imaging apparatusis subtracted, using the imaging apparatus interface.
 7. The imagingsignal extraction apparatus defined by claim 6, wherein the backgroundinformation is background fluorescence obtained from a light-fieldmicroscope.
 8. The imaging signal extraction apparatus defined by claim6, wherein subtraction of the background information comprises applyingrank-1-matrix factorization.
 9. The imaging signal extraction apparatusdefined by claim 1, wherein operation (a) comprises determining thestandard deviation of a time series of camera frames.
 10. The imagingsignal extraction apparatus defined by claim 1, wherein operation (b)comprises using a numerically simulated ballistic point-spread-functionassociated with the imaging apparatus.
 11. The imaging signal extractionapparatus defined by claim 1, wherein before operation (b), thetwo-dimensional image was thresholded to exclude residual backgroundactivity.
 12. The imaging signal extraction apparatus defined by claim1, wherein operation (b) further comprises reducing reconstructionartefacts by incorporating total-variation and sparsity constraints intothe mapping.
 13. The imaging signal extraction apparatus defined byclaim 12, wherein reducing reconstruction artefacts comprises applyingthe following equation:x _(n+1) =x _(n)(P ^(T) y/Py+λ1_(dim(x))), wherein x represents a volumeestimate, 1_(dim(x)) represents a vector of ones with same dimension asx, P represents the point-spread-function, λ represents weight of asparsity-encouraging term, and y represents background subtracted rawdata.
 14. The imaging signal extraction apparatus defined by claim 1,wherein operation (c) comprises using spatial segmentation to suppressspatial frequencies incompatible with object shapes.
 15. The imagingsignal extraction apparatus defined by claim 14, wherein the spatialsegmentation comprises: applying a bandpass filter to the threedimensional image; thresholding to exclude background artefacts, andapplying a local maximum search algorithm.
 16. The imaging signalextraction apparatus defined by claim 1, wherein operation (d) ofmapping the three-dimensional image of the candidate object with thepoint-spread-function associated with the imaging apparatus comprises:producing a sparse non-negative p×n matrix S_(i), wherein n is thenumber of object candidates, p is the number of pixels and i is theiteration number, wherein S₀ is the initial spatial forward model of thecandidate object.
 17. The imaging signal extraction apparatus defined byclaim 6, wherein obtaining background-corrected data comprises:generating a p×t matrix Y using the matrix product of S₀ and T₀, whereinT_(i) is a non-negative n×t matrix of temporal components, wherein t isthe number of time steps in the recording.
 18. The imaging signalextraction apparatus defined by claim 17, wherein T_(i) is obtained byiteratively applying an adapted Richardson-Lucy-type solver with asparsity constraint.
 19. The imaging signal extraction apparatus definedby claim 18, wherein iteratively updating the estimated spatial forwardmodel and estimated temporal components comprises: i) obtaining anupdated estimated S_(i), while keeping estimated T_(i) constant;obtaining an updated estimated T_(i), while keeping estimated S_(i)constant; and ii) iteratively repeating operation (i) until convergenceis reached, for the object candidate.
 20. The imaging signal extractionapparatus defined by claim 1, wherein the candidate object is a neuron.21. An imaging signal extraction apparatus comprising: an imagingapparatus interface; a processing device, the processing deviceoperatively coupled to the imaging apparatus interface; and a computerreadable medium comprising instructions that, when executed by theprocessing device, perform operations comprising: a) generating atwo-dimensional image from imaging information obtained from the imagingapparatus interface, thereby estimating ballistic component of theimaging information; b) generating a three-dimensional image byremapping the two-dimensional image; c) identifying a candidate objectin the three-dimensional image; d) obtaining an estimated spatialforward model of the candidate object by mapping the three-dimensionalimage of the candidate object with a point-spread-function associatedwith the imaging apparatus; e) obtaining background-corrected data byusing the estimated spatial forward model of the candidate object andestimated temporal components; and f) iteratively updating the estimatedspatial forward model and estimated temporal components untilconvergence is reached for the candidate object, thereby extracting thesignal information, wherein the imaging apparatus interface compriseshardware developed using a Miniscope platform, an implanted endoscopicGRIN relay, a sensor, and a microlens array, the microlens array beingaligned and mounted in close proximity to the sensor such that a backfocal plane and a sensor plane coincide.
 22. The imaging signalextraction apparatus defined by claim 21, wherein the microlens array isdisposed in an optical path of an image plane, the microlens array beingdisposed one focal length away from the sensor.
 23. The imaging signalextraction apparatus defined by claim 21, further comprising a holdingmember configured to hold the sensor, the holding member being elongatedby 2.7 mm when compared with the Miniscope design.
 24. A method ofextracting imaging signals, the method comprising: an imaging apparatusinterface operatively coupled to a processing device, the processingdevice performs the following operations: a) generating atwo-dimensional image from imaging information obtained from the imagingapparatus interface, thereby estimating ballistic component of theimaging information; b) generating a three-dimensional image byremapping the two-dimensional image; c) identifying a candidate objectin the three-dimensional image; d) obtaining an estimated spatialforward model of the candidate object by mapping the three-dimensionalimage of the candidate object with a point-spread-function associatedwith the imaging apparatus; e) obtaining background-corrected data byusing the estimated spatial forward model of the candidate object andestimated temporal components; and f) iteratively updating the estimatedspatial forward model and estimated temporal components untilconvergence is reached for the candidate object, thereby extracting thesignal information.